- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces LabHorizon, a dataset and benchmark for protocol-aligned action prediction in wet-lab settings, released together with a matching training set.
The benchmark (6,400-sample) has two levels. In Level 1, the model is given three rendered views of a piece of laboratory equipment, the protocol context, and the action history, and select the next action from 10 candidates. In Level 2, the model is given a protocol window and an action pool, and generate a parameterized action program of 8–13 steps. The output is parsed with Python's AST and scored on these metrics: position-wise Action Sequence Similarity (ASS), alignment-based Parameter Accuracy (PA), and their average.
The authors evaluate roughly 16 frontier and open-source models and provide an error-type analysis. L1 accuracy ranges from 0.465 to 0.555, and the best L2 final score is about 0.33. The authors further fine-tune Qwen3.6-35B-A3B with LoRA on the training set, raising L1 from 47.5% to 63.5% and L2 from 25.3% to 41.0%. Adding an Actor–Simulator–Selector agent on top, with Gemini 3.1 Pro serving as both Simulator and Selector, brings the final results to 66.5% and 45.3%, respectively.
Critical strengths
Strength 1: A protocol-grounded wet-lab action dataset. The data are derived from roughly 2,300 real wet-lab protocols. This helps narrow the gap between knowing the science and being able to carry out the experimental procedure. Because a training split is released alongside the test split, the dataset can be used both to evaluate models and to improve them
Strength 2: Experimental plans are represented as dataflow programs, and dependencies between intermediate products are checked. This is a more faithful way to evaluate long-horizon procedural planning than comparing free-form text, and the AST-based scoring could be reused for other procedural or protocol-planning tasks.
Critical weaknesses
Weakness 1: Key claims lack supporting evidence. For 3D perception: There is no ablation that removes or perturbs the images, and the text sometimes gives the answer away. For example, Figure 8 states "The next operation uses the pictured vortex mixer for protein elution," which names both the instrument and the operation. Also, 47.7% of Level 1 errors involve the "none of the above" option, which does not test perception. Furthermore, ASS compares actions position by position, so a single extra or missing step misaligns every step after it and inflates the error. PA uses monotonic alignment but ASS does not, so the two metrics are inconsistent. It seems that ASS also ignores partial-order relations between protocol steps.
Weakness 2: Limited novelty. The Level 2 setup is similar to BioPlanner, and ProBio already explored multi-view, protocol-guided lab perception. The paper's contribution is combining these ideas. The training side is standard LoRA fine-tuning. The agent adds only a small gain (L1: 63.5→66.5%, L2: 41.0→45.3%), and with no ablation
Questions for authors
-
What is the Level 1 accuracy under each of these settings? (a) text only, with the instrument name kept; (b) text only, with all instrument information removed; (c) images shuffled across samples.
-
How would the Table 1 and the error analysis change if ASS were computed with an alignment-based measure (e.g., LCS), or if Level 2 were scored against a dependency graph (DAG) of the protocol steps?
-
Where do the fine-tuning gains come from? Do they reflect genuinely better experimental reasoning, better parameter inference, or mainly the model learning the dataset's formatting and naming conventions?