- Rating
- 3
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This work proposes a laboratory robotics benchmark that organizes tasks into L1 single skills, L2 context-dependent operation sequences, and L3 complete biological experiment SOPs.
It also introduces the BioLab-HVLA method, which uses execution history and visual progress assessments to select subtasks, which are then executed by the VLA.
Critical strengths
The problem is valuable. Laboratory procedures is quite complex; even humans require some training.
Well-defined task and appropriate data setup are particularly important in this setting. The three-tiered task structure is well motivated and reasonable
Critical weaknesses
The baseline VLAs coverage is somewhat limited for comparison, in fact, a considerable number of VLAs with some memory capacity have recently emerged.
Some experiments use too few trials to reliably assess performance. Although the tasks are challenging and experiments is costly, fifteen trials per real-world task provide limited statistical evidence to rule out the influence of random variables
Questions for authors
The method involves many components, but the discussion of each component’s contribution is incomplete. For example, how much improvement comes from memory and from sub-instruction training?
Automated task generation and checker construction form a relatively complex pipeline that have a significant impact on the overall system. A clearer description of this pipeline would strengthen the paper.