acceptodds
Under review as a conference paper at ICLR 2027

Garfield: Graph Augmented Reasoning Framework for Integrated Experimental Lab Discovery

Abstract

Experimental protocols form the knowledge backbone of laboratory experiments and automation. They contain information on experimental steps, actions, materials, instruments, parameters, and execution flow, while also sharing recurring concepts across protocols. This structure and cross-protocol knowledge remain largely implicit in natural language. To address this, we introduce GARFIELD (Graph Augmented Reasoning Framework for Integrated Experimental Lab Discovery), a framework that provides experimental protocols as structured context through our proposed multi-view experimental protocol graph representation. This graph representation captures four views: hierarchical, execution, action context, and cross-protocol alignment. The hierarchical view organizes protocols into processes, steps, and actions; the execution view captures procedural flow; the action context view connects actions with their experimental objects and parameters; and the cross-protocol alignment view links equivalent actions, materials, instruments, products (outputs), and biological samples across protocols. During inference, our graph-aware GARFIELD agent dynamically interacts with the multi-view experimental protocol graph representation with a dedicated set of tools to adaptively retrieve the necessary context based on the query. Experiments show that GARFIELD outperforms language-model and retrieval baselines in retrieving experimental details (+11.5% accuracy), reasoning over questions on structural relationships (+0.326 Token-F1), detecting protocol errors (+11.6% accuracy), and resolving procedural order (+14.4% sequence EM). We also evaluate GARFIELD on protocol design, where access to its structured context improved overlap with gold protocols for experimental operations and execution flow by 27.0% and 13.6%, respectively. These results show that GARFIELD can complement large language models by providing structured experimental context for both protocol knowledge retrieval and design.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Graph Representation, Experimental Protocols, Laboratory Automation, Agentic AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ivGMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Garfield organizes experimental protocols into four graph views and uses dedicated tools to support LLMs in protocol question answering, error detection, step ordering, and protocol design, reporting improvements across multiple models and tasks.

Critical strengths
  1. Organizing protocol hierarchies, execution order, and action context into a queryable structure has a clear practical motivation.
  2. The evaluation covers five LLMs and multiple tasks, with additional analyses by domain and reasoning depth and checks of agreement with human judgments.
Critical weaknesses
  1. The main paper is insufficiently self-contained. Essential details including the graph construction procedures and specific tool functions and invocation formats are largely relegated to the appendix, making the complete method difficult to understand from the main paper alone.
  2. The main figure devotes substantial space to abstract graph structures and scientific categories without showing tool invocation, retrieval across views, or the evidence feedback process. So I think it does not convey the core makanism.
  3. Without comparisons against flat JSON, relational tables, or structured text using the same extracted information and agent, I remain unconvinced of the independent value of the multi-view graph.
  4. The authors do not evaluate the effects of multiple views with ablation study, leaving the necessity of these design choices unclear.
  5. Comparative token consumption, end-to-end latency, and offline graph construction costs are not clear while using the multi-view graph.
Questions for authors
  1. Can the authors provide a complete execution trace showing tool inputs, the graph relations accessed, returned results, and the computations performed within the tools?
  2. How does performance change when the same agent uses flat structured information, or when one or two graph views are disabled?
  3. What are the actual token consumption, runtime, and offline graph construction costs compared with the text-retrieval baselines?
3 comments
AuthorsOpenReview ↗

Response to Reviewer ivGM - Part1

We thank the reviewer for recognizing the motivation for GARFIELD as well as the breadth of evaluations. Following the raised concerns, we have extended the Methods section, providing a more self-contained framing in the main text, added an execution trace, and elaborated on token consumption details.


W1: The main paper is insufficiently self-contained.

Following the reviewer’s comment, we have expanded the Methods section, and we hope you find it more self-contained. Specifically, we added:

  1. Graph construction (Section 3.2): Added a dedicated subsection describing the complete pipeline, from protocol segmentation and hierarchical extraction to execution-flow construction, action context, cross-protocol alignment, and deterministic validation.

  2. Agent and tool interface (Box 1: Tool functions and invocation format): We added descriptions of representative graph tools, their functions, and the JSON invocation format, clarifying how the agent discovers and uses tools.

Manuscript: Added Section 3.2 to describe the graph construction and Box 1 in Section 3.3 to describe the agent tool interface.


W2: The main figure devotes substantial space to abstract graph structures and scientific categories without showing tool invocation, retrieval across views, or the evidence feedback process. So I think it does not convey the core mechanism.

We appreciate this criticism. It is true that Figure 1 provides an abstract representation of the GARFIELD framework, highlighting its utility and overall organization. However, Figure 2 provides an example of open-ended protocol question answering, demonstrating tool invocation. We believe the core mechanism of GARFIELD is two-fold, comprising both the graph structure and abstraction and the tool-calling agent. We therefore devote figures to each aspect.


Q1: Can the authors provide a complete execution trace showing tool inputs, the graph relations accessed, returned results, and the computations performed within the tools?

Thank you for making this suggestion. We added a complete illustrative execution trace of the GARFIELD agent for a BioProBench question, following the workflow in Algorithm 1, provided in Appendix G (Figure S10).

The trace shows the agent’s reasoning steps, tool calls, the graph relations accessed, returned results, and final answer. In this example, the tools search the step text, follow the NEXT and HAS_ACTION relations, and retrieve the parameters of the identified action.

Manuscript: Added Figure S10.

AuthorsOpenReview ↗

Response to Reviewer ivGM - Part2

W3 / W4 / Q2: How does performance change when the same agent uses flat structured information, or when one or two graph views are disabled?

Thank you for raising this point. GARFIELD's multi-view graph provides a unified representation for querying experimental relationships across procedural levels and protocol boundaries, and the agent’s tools are tightly connected to the representation. Flat JSON or structured text can also store extracted steps, actions, and parameters. However, accessing dependencies, material transformations, and relationships across protocols requires reconstructing these connections during retrieval or explicitly implementing them through additional queries. GARFIELD instead makes these relationships directly accessible through its interconnected views and specialized graph operations. The primary motivation is therefore structured and direct access to procedural relationships, rather than a claim that graphs are inherently more expressive than alternative representations.

We acknowledge that we have not performed a controlled ablation replacing the graph with flat structured information or selectively disabling individual graph views. Consequently, our current results do not isolate the contribution of each view. Nevertheless, the experiments provide task-level evidence consistent with the benefits of structured relational access. In Table 2, Retrieving@5 improves child-step ordering from 42.04% to 61.64%, but leaves top-step ordering nearly unchanged (25.68% versus 25.70%). GARFIELD achieves 75.10% and 41.38%, respectively. These results suggest that access to hierarchical and execution relationships may be particularly valuable for higher-level procedural questions.


W5 / Q3: What are the actual token consumption, runtime, and offline graph construction costs compared with the text-retrieval baselines?

We now report token usage in Appendix D.2. During inference, token usage for retrieving@3 and retrieving@5 depends on the length of the retrieved protocols, while GARFIELD’s usage depends on the number of tool calls and the information returned by each call. Averaged across BioProBench and BioPIE, retrieving@3 uses 6.3k tokens per query, retrieving@5 uses 10k, and GARFIELD uses 9.1k. GARFIELD therefore achieves better performance than retrieving@5 while using fewer tokens per query. The average inference time is 5.8 s per query for retrieving@3, 7.6 s for retrieving@5, and 10.1 s for GARFIELD.

Token usage during one-time offline graph construction depends on protocol length. Constructing the Hierarchical, Execution, and Action-Context Views, including preprocessing and excluding entity normalization, uses an average of 20k input tokens per protocol, or approximately 25–30k total tokens including model outputs. Entity normalization for the Cross-Protocol Alignment View then uses approximately 75 tokens per local action or experimental object. Each batch call also includes approximately 360 instruction tokens, shared across the entities processed in that batch. We used Claude Opus-4.8 as the core model for all these measurements.

Manuscript: We added these measurements to Appendix D.2.

Thank you for the clarifications and additional cost measurements. However, comparisons with equivalent flat structured representations and graph-view ablations remain missing, leaving the contribution of the multi-view design insufficiently established. I therefore maintain my original score of 1 (clear rejection).

Reviewer_i5qrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GARFIELD converts wet-lab protocols into a four-view graph (process, step, and action hierarchy, execution flow, action context, canonical nodes) via Opus-4.8 prompts with verbatim checks. A ReAct agent queries it through ToolRAG over 23 tools. Across five LLMs, it beats closed-book and retrieval baselines on BioProBench (PQA, ERR, ORD gain +11.5, +11.6, +14.4 pp vs. retrieving@5) and BioPIE (EM 25.64% to 58.36%).

Critical strengths

Steps and actions must match verbatim spans under deterministic checks (App. A.1–A.3), and documented schema, tools, and prompts keep answers grounded.

BioPIE QA gains hold across five LLMs, both judges (+27.3 and +24.4 pp vs. retrieving@5), and all hop depths, with strong human / judge agreement (kappa 0.856–0.906, Table S10).

Critical weaknesses

BioProBench ORD and ERR may test lookup rather than reasoning. Source protocols look indexed (Sec. 6, Fig. 2), where Find_Matching_Step_Order returns steps "in the order recorded in that protocol" and Find_Matching_Steps can return the uncorrupted step (Table S14). ORD averages 1.59 tool calls (App. D), while retrieval baselines lack step-order lookup. I want a leave-source-out test and per-tool counts.

Nothing ablates views, tools, or ToolRAG, so graph value is mixed with the agent loop. Single-shot top-3 and top-5 retrieval baselines face an up-to-16-step, 23-tool agent (five for ordering, Table S14), with no agentic text-RAG or GraphRAG baseline and no build / query cost. In protocol design, GARFIELD gets six protocols plus summaries versus three (p. 6), yet IoU(Op) moves only from 0.203 to 0.258.

Minor:

Graph quality: 20 protocols checked manually without error rates (p. 14), and no variance / CIs. Prompts keep {Example} and {Schema} placeholders (Figs. S3–S7), with no code release statement. Positioning: discuss Process Execution Graphs (2021) on coreference ("supernatant"), Graph-CoT (2024) on graph-tool agents, and BioPIE's KG-based QA (2026).

Questions for authors

Is each question's source protocol in the graph? How do BioProBench ORD and ERR results (GARFIELD and retrieval) change when it is held out?

On one backbone, how does the same agent loop and budget perform with raw-text step search instead of the graph?

1 comment
AuthorsOpenReview ↗

Response to Reviewer i5qr

Thank you so much for taking the time reviewing the paper, recognizing the strengths of GARFIELD as well as raising the remaining concerns which we relate to in detail here.


W1 / Q1: Is each question's source protocol in the graph? How do BioProBench ORD and ERR results (GARFIELD and retrieval) change when it is held out?

Thank you so much for raising this question. Yes, each question’s source protocol is included in GARFIELD’s graph and in the corpus available to the retrieval baselines. Our evaluation tests how effectively each method uses documented protocols to recover step order and identify errors. This reflects laboratory tasks such as checking instructions and reproducing an experiment, which still requires reasoning and understanding. GARFIELD exposes this information through its graph retrieval tools, while the baselines provide protocol text. With access to the same corpus, GARFIELD improves ERR accuracy by 11.6% and ORD sequence EM by 14.4% over retrieving@5 (Table 1).

Holding out the source protocol evaluates transfer to unseen experimental procedures. Without the original guide, multiple execution sequences may be scientifically valid. Consequently, evaluating against the withheld protocol’s exact sequence or error annotations may penalize valid alternatives rather than measure scientific correctness. For this reason, ORD and ERR evaluation focuses on recovering the original step order and detecting errors in instructions when the source protocol is available. This follows the benchmark setting defined in BioProBench (Liu et al., 2026).

As mentioned in Section 6, generalization without access to source protocol remains future work. We have, however, explored how knowledge from existing protocols can help generate new procedures in our protocol-design experiments (Section 5.3). Compared with the protocol-context baseline, GARFIELD improves operation IoU from 0.203 to 0.258 and execution similarity from 0.348 to 0.396 (Table 5).

Manuscript: We revised Section 4 to clarify that source protocols are included in the graph and retrieval corpus for both BioProBench and BioPIE datasets. We also improved the discussion of limitations and future work in Section 6.

Reference

[1] Liu, Y., Lv, L., Zhang, X., Wang, J., Yuan, L., & Tian, Y. BioProBench: A Corpus and Benchmark for Biological Protocol Reasoning in Autonomous Science. In Forty-third International Conference on Machine Learning (ICML 2026).


W2 / Q2: Nothing ablates views, tools, or ToolRAG, so graph value is mixed with the agent loop [..] On one backbone, how does the same agent loop and budget perform with raw-text step search instead of the graph?

We agree that current baselines do not isolate graph contribution from the agentic loop. GARFIELD's multi-view graph provides a unified representation for querying experimental relationships across procedural levels and protocol boundaries, and the agent’s tools are tightly connected to the representation. The primary motivation is therefore structured and direct access to procedural relationships, rather than a claim that graphs are inherently more expressive than alternative representations. We see the main contribution of this approach in providing a structured foundation for language models to access structured protocols and use the experimental structure that is implicit in their instructions. GARFIELD makes this structure explicit by recording which actions belong to each step and process, how they are carried out, and which experimental objects and conditions are involved.

Our existing comparisons provide supporting evidence for this combined approach. Averaged across the five backbones, GARFIELD achieves 41.38% sequence EM for top-step ordering, compared with 25.70% for retrieving@5 (Table 2). Figure 2 and runtime traces show that retrieving the correct source protocol does not always lead to the correct answer: retrieving@5 retrieves it at rank 2 but answers incorrectly, whereas GARFIELD identifies the relevant action–instrument relationship, demonstrating the contribution of structured access.

Reviewer_9WoPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents GARFIELD, a framework that represents experimental protocols as graphs. The framework has four views: hierarchical execution, action context, and cross-protocol alignment. The paper also presents a GARFIELD agent that interacts with these protocol graphs using a set of tools to complete tasks such as question answering, or identifying errors in a protocol. Experiments compare GARFIELD to language model and retrieval baselines showing improvements on retrieval, question answering, detecting errors, and protocol design.

Critical strengths

Understanding of scientific protocols is an important problem, and the symbolic graph representation seems like a useful approach. The experiments show improvements across a number of tasks.

Critical weaknesses

The methodology and novelty need a bit clearer justification. The main contributions seem to be in the graph representation. Graph construction uses a prompted LLM. The paper should clarify what is it's contribution relative to prior work on protocol representations and graph retrieval systems.

Some relevant prior work is missing, for example, WLP, the WNUT wet-lab extraction shared task, and X-WLP's process-level graphs are closely related. Comparing to their representations would help to establish what GARFIELD is adding that is new. https://arxiv.org/pdf/2101.10244 https://arxiv.org/pdf/1805.00195

Graph correctness is not quantitatively evaluated. The authors manually verified 20 protocols, but don't report any quantitative metrics against human annotations.

Questions for authors

What capabilities does the GARFIELD representation add beyond existing graph-based protocol representations, such as X-WLP?

Can you evaluate graph extraction against human annotations? What is the accuracy or precision/recall of the extracted graphs?

1 comment
AuthorsOpenReview ↗

Response to reviewer 9WoP - Part1

Thank you for recognizing the importance of the problem GARFIELD tackles and the value of the suggested approach, as well as the constructive feedback that helped improve the manuscript. Below, we provide detailed answers to the raised concerns.


W1 / W2 / Q1: What capabilities does the GARFIELD representation add beyond existing graph-based protocol representations, such as X-WLP?

Thank you so much for raising this point and for suggesting these references. We agree that the comparison should be clearer. WLP and the WNUT 2020 wet-lab protocol corpus annotate experimental actions, the materials and instruments each action uses, and the parameters specified for it. X-WLP extends these works by tracking which actions use or process the same materials across instructions and recording which actions occur before others. GARFIELD adds entities and relations that address three further aspects of protocol representation.

  1. Internal hierarchical structure understanding of protocols (Hierarchical View); protocols can include several processes, each consisting of steps that contain multiple actions. These levels are important because the same action can appear in different parts of an experiment, and its role depends on the step and larger process it belongs to. For example, a DNA-isolation protocol has “Add lysis buffer and mix” during cell lysis and the step “Mix and store at −20°C for at least one hour” during DNA precipitation. Both steps include mixing, but it serves a different purpose in each process. X-WLP connects actions with their associated experimental objects but does not capture which step and process each action belongs to. GARFIELD addresses this with its Hierarchical View, linking each action to its step and each step to its experimental process.

  2. Understanding protocol execution requires knowing relations between processes, steps, actions, and conditions (Execution View). X-WLP’s “succ” relations record temporal precedence, but do not represent these execution patterns. GARFIELD adds dedicated relations for parallel execution, repetition, and conditional paths within its Execution View.

  3. Protocols are connected through the experimental actions and objects they share (Cross-Protocol Alignment View.). X-WLP classifies actions and objects into predefined categories but does not connect matching actions and objects across protocols. GARFIELD addresses this gap through its Cross-Protocol Alignment View. It links matching actions and objects across protocols to shared entries, even when their names differ. Each use retains its context, participating objects, and parameters.

Manuscript: We revised Section 2 to include the suggested references and clarify GARFIELD’s differences from prior representations, and Section 3.1 to better explain the motivation for the Hierarchical and Execution Views.


W3 / Q2: Can you evaluate graph extraction against human annotations? What is the accuracy or precision/recall of the extracted graphs?

To address this question and quantitatively evaluate graph extraction for each view, we created a ground-truth set. A PhD student with backgrounds in computer science, electrical engineering, and biomedical engineering was provided with the original protocol texts and annotation guidelines to extract the nodes and relationships for each of the four views. We then compared GARFIELD's extracted graphs with the ground-truth graphs.

We report precision, recall, and F1 for each view, evaluated by comparing the nodes and relations of the graphs (defining a false positive as a node or relation present in GARFIELD but missing from the ground truth, and a false negative as a ground-truth node or relation missing from GARFIELD).

The results are presented in the following table and included in Appendix A.5 of the revised manuscript. These results show strong agreement between GARFIELD's constructed graphs and the ground truth, with an overall F1 of 86.9%.

ViewPrecisionRecallF1
Hierarchical94.5%96.9%95.7%
Execution84.9%84.0%84.5%
Action Context87.8%72.7%79.6%
Cross-Protocol Alignment90.3%88.6%89.4%
Overall89.9%84.1%86.9%

Manuscript: These results are added as Appendix A.5 in the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.