acceptodds
Under review as a conference paper at ICLR 2027

ProtoWorld: A Benchmark for World Modeling from Laboratory Protocols

Abstract

Laboratory protocols specify how to perform and reproduce experiments. Developing and executing them takes expertise and time, with some experimental workflows spanning months or even years. If AI agents could automate these procedures, they could save researchers time and effort. To do so, they would first need to model how experimental actions change the contents of laboratory containers, such as tubes. Thus, we introduce ProtoWorld, a benchmark for evaluating this aspect of world modeling from laboratory protocols. We give a model the instructions from the start of a protocol through a selected step and ask either how much liquid a container holds or which experimental substances (reagents) it contains after that step. If the volume cannot be determined from the instructions, the model must answer indeterminate. We report a ProtoWorld Full set of 100 manually annotated questions at 82 selected steps across 43 protocol records, including a ProtoWorld Hard subset of 40 questions manually labeled as hard. We use a large language model (LLM) to extract operations such as additions, transfers, and removals from the protocol. Then, we manually review these operations, and apply a deterministic program to these operations to calculate the reference answers. Across thirteen model configurations, the highest overall accuracy is 64.7% on ProtoWorld Full and 30.0% on ProtoWorld Hard. We hope ProtoWorld will help researchers evaluate and improve the world-modeling capabilities that agents need to design, refine, and execute laboratory protocols, ultimately helping accelerate scientific discovery.

Primary area:
datasets and benchmarks
Keywords:
large language models, procedural reasoning, state tracking, reasoning under incomplete information, scientific benchmarks, laboratory protocols

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bqf3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ProtoWorld, a benchmark for evaluating the ability of LLMs to track laboratory container states from natural-language experimental protocols. The benchmark includes 100 questions from 43 real laboratory protocols, covering liquid volume calculation, reagent identification, and recognition of indeterminate states. The authors use an LLM to extract structured operations, manually review them, and compute reference answers using a deterministic program. Experiments on 13 model configurations show that current LLMs still struggle with reagent tracking, recognizing missing information, and maintaining consistent states across experimental steps.

Critical strengths
  1. The paper is easy to follow. The motivation, benchmark construction, and evaluation procedure are clearly presented.

  2. The paper addresses a practical problem in scientific AI agents. Using real laboratory protocols provides a meaningful setting for evaluating how LLMs interpret experimental instructions and track changes in container states.

  3. The benchmark considers both state reconstruction and the recognition of insufficient information. The evaluation across 13 model configurations provides useful observations about current model limitations.

Critical weaknesses
  1. The benchmark is relatively small, which limits the reliability of its findings. ProtoWorld Full contains only 100 questions, while ProtoWorld Hard is a subset of 40 questions. In particular, only 16 questions in Full and eight in Hard evaluate indeterminate states. Some state-transition depth groups contain just one or two samples. Such limited coverage makes it difficult to draw reliable conclusions about model performance across different experimental operations and reasoning difficulties.

  2. The evaluation does not clearly distinguish instruction understanding from world modeling. Models are required to interpret natural-language protocols and track container states simultaneously. An incorrect answer could therefore result from misunderstanding an instruction rather than failing to maintain the correct state. Since the authors already have manually reviewed structured operations, evaluating models with these operations as input would help clarify whether the observed errors actually reflect limitations in state tracking.

  3. The experimental findings provide limited new insights beyond existing studies. The reported difficulties in tracking reagents, recognizing missing information, and maintaining states across multiple steps are closely related to previously studied problems in procedural reasoning, state tracking, and abstention. Moreover, the difference between volume and reagent-set accuracy may partly result from different task difficulties and scoring criteria. The paper does not sufficiently explain what new understanding these observations provide about LLM world modeling.

  4. The reliability of the benchmark annotations needs further clarification. The reference answers depend on structured operations initially extracted by an LLM and subsequently reviewed by the authors. However, the paper does not provide quantitative evidence of extraction accuracy, operation coverage, or annotation agreement. This raises concerns about the reliability of the reference states used for evaluation.

Questions for authors
  1. The benchmark contains relatively few questions, especially for indeterminate states and deeper state transitions. How do the authors ensure that the reported findings are representative and statistically reliable?

  2. Since the authors already have manually reviewed structured operations, could they evaluate models directly on these operations? This would help determine whether the observed errors mainly arise from interpreting natural-language instructions or from tracking state changes.

  3. What are the main new insights provided by ProtoWorld compared with existing state-tracking and procedural-reasoning benchmarks? In particular, how can the authors distinguish genuine differences in state-tracking ability from differences in task difficulty or scoring criteria?

  4. How was the correctness and completeness of the extracted operations verified? Was any independent annotation or agreement analysis conducted to establish the reliability of the reference answers?

Reviewer_k2VwOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ProtoWorld, a benchmark for testing whether language models can track container contents from written laboratory protocols. Given the instructions up to a selected step, models must predict a container’s liquid volume or the reagents it contains. They must also recognize when the instructions do not provide enough information to determine the volume. The benchmark contains 100 manually annotated questions from 43 protocol records, including a 40-question hard subset. Reference answers are built using LLM-extracted operations, manual review, and a deterministic program that tracks changes in container contents.

Critical strengths
  • The benchmark tests whether models can track container contents and recognize when information is missing. These are practical requirements for agents that assist with experiments. Using real protocols makes this a meaningful setting for evaluating those abilities.
  • The paper separately measures volume calculations, reagent identification, and recognition of unknown volumes. For example, Claude Opus 5 achieves 83.3% volume accuracy but only 39.2% reagent-set accuracy. This breakdown makes the benchmark informative: success on one part of the task does not imply reliable tracking of the full container state.
  • An unspecified addition can make a volume unknown, while later emptying the container can make it known again. Handling this correctly requires following how uncertainty changes across steps. This is a useful aspect of the benchmark because an experimental assistant needs to recognize both when information is missing and when that missing information no longer matters.
Critical weaknesses
  • The dataset contains 100 questions from 43 protocol records, but the paper does not clearly explain how protocols and questions were selected or how the hard subset was defined. It also does not report agreement between independent annotators on the reference answers. Since the benchmark is the main contribution, these gaps make it difficult to judge whether the findings extend beyond this small collection.
  • Incorrect answers could come from misunderstanding instructions, identifying the wrong container, mismatching reagent names, or losing track of state. The paper acknowledges some of these possibilities, but provides no controlled comparisons using simplified instructions, structured operations, or a tool-assisted baseline. Such comparisons would help explain what causes the low scores. Without them, the evidence about models’ world-modeling limitations remains limited.
  • The benchmark uses simplified rules, such as treating washing as completely emptying a container and clearing its reagent set. These rules make scoring consistent, but the paper does not establish whether they adequately represent the included experimental procedures. - This limits confidence that performance on the benchmark reflects the abilities needed in actual laboratory work. Each model is evaluated with one shared prompting setup. The paper does not test whether explicit state tables, worked examples, or access to a simple tracking tool would substantially improve performance. This matters because laboratory agents could use these aids, and the current results leave unclear how much difficulty remains under a well-designed agent setup.
Questions for authors
  • How were the protocols, query points, and hard questions selected? Were these choices fixed before examining model outputs? This would help assess whether the reported difficulty depends on how the collection was chosen.
  • How many annotators independently checked the reference answers, what relevant expertise did they have, and how were disagreements resolved? How many answers were changed following the model-assisted reference review?
  • Can the existing outputs distinguish failures in interpreting instructions or matching reagent names from failures in updating container states? Do you have results using explicit state tables or a deterministic tracking tool? This evidence would affect how strongly I interpret the low scores as limitations in world modeling.
Reviewer_Tm93OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ProtoWorld tests whether a model can track laboratory container contents while reading a wet-lab protocol. Given the text up to a query step, the model reports a container's liquid volume, its reagent set, or "indeterminate". Full has 100 questions at 82 query points from 43 protocols.io records (50 volume, 34 reagent-set, 16 indeterminate). Hard is a 40-question subset labelled hard by the authors (Table 1; L87-92). References come from LLM-extracted, author-revised operations, a deterministic ledger and manual corrections (L157-160, L191-198). Thirteen configurations, three runs each, reach at best 64.7% overall on Full and 30.0% on Hard (Tables 2-3).

Critical strengths
  1. A well-posed task that scores answerability as its own axis. It has an explicit indeterminate answer, judge-free scoring (L127-150, L796-804), and separate false-abstention and numerical-commitment measures (Table 4). This matters because no cited protocol benchmark couples state reconstruction with recognition of missing information (Section 5). Recovery after emptying (Fig. 1; L146-150) is a good idea, though not yet measured (Weakness 2).
  2. Exact reporting, and an abstention effect that survives on uncontested items. The aggregates in Tables 2, 3 and 6 recompute from their parts, and the released inputs reproduce the paper's counts and statistics (public/atlas.json). Seven configurations give a number in every run on the two clearly indeterminate Hard items (CB-0239, CB-0730; Table 4). Claude Opus 5 gives a number on 21 of 24 non-Hard indeterminate responses (Table 4 minus Hard). This matters because high answerable accuracy does not imply abstention. For GPT-6 Astra and GPT-5.5, however, the non-Hard rate is only 11 and 10 of 24.
Critical weaknesses
  1. The references, which carry every number, are unaudited, and the checkable part does not hold up. An LLM-assisted review of questions that several models answered identically led to removals and restorations decided by the authors (L491-492). Section 2.2 omits it (L157-160, L195-196), and there are no counts, no named extraction LLM and no agreement study. Under the paper's rules (L141-150), only two of the eight Hard indeterminate references are clearly indeterminate from the released prefixes (public/questions/CB-*.json):

    • CB-0092 asks for well A1 of the "384-well plate". For the plate being prepared, steps 4, 6 and 9 add 5 µL, 1000 nL and 25 µL to each of its 192 wells, which gives 31 µL; the only other 384-well plate in the prefix is step 6's primer source plate.
    • CB-0346 to CB-0348 target a "(default plate)" absent from the prefix. Both tubes the prefix describes are determinate at steps 34-35: one is emptied at step 26; the other is washed at step 33, cleared at step 34 and given 15 µL at step 35 (L142, L148-149).
    • CB-0153 (its query step, "Repeat Steps 48 and 49", points outside the 0-26 numbering) and CB-0637 (a hypothetical variant at step 54) are debatable.

    The Hard recall and commitment columns (Tables 3-4), L368-369, and the Opus/Llama tie at 30.0% (L316-317) rest on these labels. Nothing else can be audited. The paper describes construction records, alias map, prompt and scorer as released (L154-155, L220, L501-502, L792-794), but the repository holds question inputs only, and its README states that reference answers, aliases, review records and model outputs are excluded. This is consequential because the benchmark is the contribution.

  2. The headline subset and the distinctive construct are not substantiated. Hard has no stated criterion, and the paper does not say whether labels were assigned before model results (L22, L88-89, L207). One protocol supplies 7 of its 40 items (public/atlas.json). Three numbers show that Hard carries almost all of the benchmark's signal:

    • Table 2 minus Table 3 puts GPT-6 Astra at 88.3% overall on the 60 non-Hard questions.
    • Under common weights, Grok-4.1-fast and Llama-3.3-70B lose only 4.1 and 4.5 points from Full to Hard, against 16.7 to 36.0 for the others (Table 6).
    • Answering indeterminate everywhere scores 20% on Hard, above GLM-5.3 and Nemotron-3-Ultra (Table 3).

    The distinctive construct has three gaps:

    • The paper says it tests recognition of a volume becoming determinate again (L464-467), but reports no count or accuracy for such questions; Appendix E concedes that its depth bins ignore emptying (L853-855).
    • The case study promised in contribution 3 (L116-117) does not appear.
    • The released items apply emptying inconsistently: CB-0155 is determinate after "remove the remaining supernatant", while CB-0346 to CB-0348 are not.

    This is consequential because the 30.0% headline and the claimed novelty rest on these points.

Questions for authors

Clarification (answers could change my assessment):

  1. Please derive the eight Hard indeterminate references under the stated rules, and rescore Tables 3-4 and the abstract if any labels change. In particular:
    • CB-0092: which plate is meant, and why the answer is not 31 µL;
    • CB-0346 to CB-0348: what "(default plate)" A1 and A2 denote, and how the prompt named them;
    • CB-0153 and CB-0637.
  2. Which LLM produced the operations and flagged candidate errors? How many annotators were there, with what background, and what fraction of operations changed? How many questions were flagged, corrected, removed or restored, and in which direction relative to the model-majority answer?
  3. What was the criterion for the hard label, who applied it, and were model outputs available at the time?
  4. How many questions involve a volume that becomes determinate again, and how do models score on them? Where is the contribution-3 case study?
  5. Did the evaluated prompts contain the "%s %s" placeholders found in 24 of the 100 released contexts (e.g. public/questions/CB-0196.json)? Do the affected reagent-set references depend on the missing tables?
  6. On what dates and through which endpoints was each configuration run?

Suggestions:

  • Release the references, alias map, scorer and prompt now, or correct the present-tense statements.
  • Report protocol-clustered intervals and a paired comparison. Each "highest" claim rests on one answer: 181 vs 180 of 252 (L266-267), and 36 vs 35 of 96 (L315-316).
  • Add an always-indeterminate row, and an extract-then-ledger or gold-operations condition.
  • Report how many reagent-set errors are naming mismatches (L313-314).
  • Discuss pretraining exposure.
  • Redact the third-party e-mail addresses that appear in 10 released contexts (cf. L503-504).

Rating rationale.

Not a clear rejection: the task is sound and new, the reporting is exact, and the abstention effect holds on uncontested items. Every defect can be fixed by relabelling, rescoring stored outputs and documenting the construction process. Not an acceptance: the benchmark is the contribution, and the only references that can be checked largely fail the paper's own rules. What would move this to a weak acceptance: derivations or corrected labels with rescored tables, the review counts, a Hard criterion fixed before model results, and a recovery analysis.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.