acceptodds
Withdrawn from ICLR 2027

Pushing the Limits of Laboratory 3D Perception and Long-Horizon Planning via Protocol-Aligned Action Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

AI Scientist systems are moving toward physical laboratory operation, which requires asset understanding, protocol alignment, and long-horizon state tracking beyond general scientific knowledge. We introduce LabHorizon, a training and evaluation framework for Protocol-Aligned Action Prediction. Level 1 predicts the next action from multi-view laboratory assets and protocol context; Level 2 generates parameterized action sequences from protocol windows and action pools. The release contains 6,400 samples: 3,000 training and 200 test samples per level, spanning 108 Level-1 assets and 2,342 unique wet lab protocols. Evaluations across major model families reveal persistent failures in asset-conditioned action selection, numerical parameters, action order, and variable dependencies. We train a Qwen3.6-35B-A3B domain model and build an Actor-Simulator-Selector agent. Training improves Level 1 accuracy from 47.5% to 63.5% and Level 2 final score from 25.34% to 41.00%; the agent further improves them to 66.5% and 45.32%. LabHorizon thus provides diagnostic evaluation and learnable supervision for laboratory action prediction.

Primary area:
datasets and benchmarks
Keywords:
llm, agent, benchmark, dataset

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_xVuuOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LabHorizon, a dataset and benchmark for protocol-aligned action prediction in wet-lab settings, released together with a matching training set.

The benchmark (6,400-sample) has two levels. In Level 1, the model is given three rendered views of a piece of laboratory equipment, the protocol context, and the action history, and select the next action from 10 candidates. In Level 2, the model is given a protocol window and an action pool, and generate a parameterized action program of 8–13 steps. The output is parsed with Python's AST and scored on these metrics: position-wise Action Sequence Similarity (ASS), alignment-based Parameter Accuracy (PA), and their average.

The authors evaluate roughly 16 frontier and open-source models and provide an error-type analysis. L1 accuracy ranges from 0.465 to 0.555, and the best L2 final score is about 0.33. The authors further fine-tune Qwen3.6-35B-A3B with LoRA on the training set, raising L1 from 47.5% to 63.5% and L2 from 25.3% to 41.0%. Adding an Actor–Simulator–Selector agent on top, with Gemini 3.1 Pro serving as both Simulator and Selector, brings the final results to 66.5% and 45.3%, respectively.

Critical strengths

Strength 1: A protocol-grounded wet-lab action dataset. The data are derived from roughly 2,300 real wet-lab protocols. This helps narrow the gap between knowing the science and being able to carry out the experimental procedure. Because a training split is released alongside the test split, the dataset can be used both to evaluate models and to improve them

Strength 2: Experimental plans are represented as dataflow programs, and dependencies between intermediate products are checked. This is a more faithful way to evaluate long-horizon procedural planning than comparing free-form text, and the AST-based scoring could be reused for other procedural or protocol-planning tasks.

Critical weaknesses

Weakness 1: Key claims lack supporting evidence. For 3D perception: There is no ablation that removes or perturbs the images, and the text sometimes gives the answer away. For example, Figure 8 states "The next operation uses the pictured vortex mixer for protein elution," which names both the instrument and the operation. Also, 47.7% of Level 1 errors involve the "none of the above" option, which does not test perception. Furthermore, ASS compares actions position by position, so a single extra or missing step misaligns every step after it and inflates the error. PA uses monotonic alignment but ASS does not, so the two metrics are inconsistent. It seems that ASS also ignores partial-order relations between protocol steps.

Weakness 2: Limited novelty. The Level 2 setup is similar to BioPlanner, and ProBio already explored multi-view, protocol-guided lab perception. The paper's contribution is combining these ideas. The training side is standard LoRA fine-tuning. The agent adds only a small gain (L1: 63.5→66.5%, L2: 41.0→45.3%), and with no ablation

Questions for authors
  1. What is the Level 1 accuracy under each of these settings? (a) text only, with the instrument name kept; (b) text only, with all instrument information removed; (c) images shuffled across samples.

  2. How would the Table 1 and the error analysis change if ASS were computed with an alignment-based measure (e.g., LCS), or if Level 2 were scored against a dependency graph (DAG) of the protocol steps?

  3. Where do the fine-tuning gains come from? Do they reflect genuinely better experimental reasoning, better parameter inference, or mainly the model learning the dataset's formatting and naming conventions?

Reviewer_1bDiOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

LabHorizon evaluates protocol-aligned action prediction at two levels: selecting the next action from multi-view instrument images and context, and generating parameterized action sequences with variable dependencies. The authors provide training and test data, evaluate models using selection accuracy and structured sequence metrics, and investigate domain fine-tuning and an Actor–Simulator–Selector agent.

Critical strengths

Connecting computational planning with wet-lab execution and feedback is a valuable application direction.

Critical weaknesses

No concrete execution embodiment or deployment interface is established. How the symbolic outputs would drive real hardware or be used in laboratory practice remains unclear.

Questions for authors
  1. Do the inputs uniquely determine the reference answers? For example, level1_test_000005 does not explicitly provide several target parameters used to distinguish candidates. Please supply the actual evaluation prompts and identify the input evidence for each decisive parameter, including any protocol context omitted from the released records.
  2. Are source protocols isolated across training and testing, including across levels? level1_test_000005 closely resembles training examples such as level1_train_000468. Please provide protocol identifiers, split overlaps, and semantic deduplication results, distinguishing intended asset sharing from protocol-stage reuse and evaluating generalization to unseen protocols.
  3. ASS compares actions at fixed positions, potentially penalizing shifted sequences or interchangeable independent steps. How does evaluation recognize multiple valid orders and dependency-equivalent plans? Please assess agreement with partial-order constraints or expert judgments of plan validity.
  4. What evidence establishes the contribution of visual input? Please compare text-only, single-view, three-view, and shuffled-image conditions. Since L2 is text-only, how is improved L1 perception connected to improved planning rather than two independently evaluated tasks?
  5. Could a complete L1-to-L2 pipeline be deployed directly on a robot or another execution platform? Please specify the intended embodiment, how symbolic actions map to executable instrument commands, and what additional perception, feedback control, safety checking, and failure-recovery components are required.
Reviewer_ucYGOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a benchmark for protocol-aligned action prediction for laboratory operations. Two different “levels” are proposed, the first needing to choose the correct next action from context (including three views of an object), and the second aims to propose an action plan from the text-only context. Results show that out-of-the-box, recent LLMs struggle to accurately predict the correct actions, and fine-tuning a Qwen model improves accuracy. Additionally, an agent consisting of multiple LLMs is proposed, that further improves accuracy.

Critical strengths
  • This paper explores a relevant problem, as agentic laboratory systems seem a promising direction.
  • The appendix is extensive and lists multiple examples, making it easier for readers to see actual examples of inputs and outputs to the model.
Critical weaknesses
  • The paper’s main contribution and takeaway are hard to identify, as it presents a dataset (with two distinct so-called levels), a fine-tuning result, and an agent design that further improves upon the fine-tuned model. Each of these shares the nine pages, and thus none of them are treated in depth with the appropriate amount of experimentation or analysis.
  • Related to the point above, the writing is quite dense. Many sentences list multiple items, without providing intuition. For example, lines 186-191 list different items the validators check. However, there is no explanation given as to why these items were chosen or how each is validated. As a whole, a majority of the text seems to describe the method, but does not provide insights into why certain choices were made.
  • The paper also uses several terms relatively loosely, and overstates its contributions. For example, the title says it’s pushing the limits of 3D perception. However, the 3D perception aspect of the method is simply passing three different 2D views of an object to a model. There is no 3D reconstruction, or any explicit depth or pose information. Another example is the term long-horizon: if I understood correctly, in Level 2, only 8-13 actions are to be predicted, which does not seem to be long-horizon. And another example is the simulator Section 3.6, which in their case seems to refer to an LLM prompted to predict the outcome, not a simulator in the traditional sense.
  • Many of the main design choices are not ablated. For example, it is unclear whether the three images in Level 1 are even needed. Would text-only, or only a single view, suffice? Similarly, the model is trained on an even 50/50 split of both levels. Would it perform better with a different split? How do the two levels complement each other during training? What if you trained an individual model for each level?
Questions for authors
  • Can you explain the role of the images in Level 1? Could the object description simply be input as text instead?
  • What would happen if the model was trained on different splits between the two levels?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.