- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Garfield organizes experimental protocols into four graph views and uses dedicated tools to support LLMs in protocol question answering, error detection, step ordering, and protocol design, reporting improvements across multiple models and tasks.
Critical strengths
- Organizing protocol hierarchies, execution order, and action context into a queryable structure has a clear practical motivation.
- The evaluation covers five LLMs and multiple tasks, with additional analyses by domain and reasoning depth and checks of agreement with human judgments.
Critical weaknesses
- The main paper is insufficiently self-contained. Essential details including the graph construction procedures and specific tool functions and invocation formats are largely relegated to the appendix, making the complete method difficult to understand from the main paper alone.
- The main figure devotes substantial space to abstract graph structures and scientific categories without showing tool invocation, retrieval across views, or the evidence feedback process. So I think it does not convey the core makanism.
- Without comparisons against flat JSON, relational tables, or structured text using the same extracted information and agent, I remain unconvinced of the independent value of the multi-view graph.
- The authors do not evaluate the effects of multiple views with ablation study, leaving the necessity of these design choices unclear.
- Comparative token consumption, end-to-end latency, and offline graph construction costs are not clear while using the multi-view graph.
Questions for authors
- Can the authors provide a complete execution trace showing tool inputs, the graph relations accessed, returned results, and the computations performed within the tools?
- How does performance change when the same agent uses flat structured information, or when one or two graph views are disabled?
- What are the actual token consumption, runtime, and offline graph construction costs compared with the text-retrieval baselines?
3 comments
Response to Reviewer ivGM - Part2
W3 / W4 / Q2: How does performance change when the same agent uses flat structured information, or when one or two graph views are disabled?
Thank you for raising this point. GARFIELD's multi-view graph provides a unified representation for querying experimental relationships across procedural levels and protocol boundaries, and the agent’s tools are tightly connected to the representation. Flat JSON or structured text can also store extracted steps, actions, and parameters. However, accessing dependencies, material transformations, and relationships across protocols requires reconstructing these connections during retrieval or explicitly implementing them through additional queries. GARFIELD instead makes these relationships directly accessible through its interconnected views and specialized graph operations. The primary motivation is therefore structured and direct access to procedural relationships, rather than a claim that graphs are inherently more expressive than alternative representations.
We acknowledge that we have not performed a controlled ablation replacing the graph with flat structured information or selectively disabling individual graph views. Consequently, our current results do not isolate the contribution of each view. Nevertheless, the experiments provide task-level evidence consistent with the benefits of structured relational access. In Table 2, Retrieving@5 improves child-step ordering from 42.04% to 61.64%, but leaves top-step ordering nearly unchanged (25.68% versus 25.70%). GARFIELD achieves 75.10% and 41.38%, respectively. These results suggest that access to hierarchical and execution relationships may be particularly valuable for higher-level procedural questions.
W5 / Q3: What are the actual token consumption, runtime, and offline graph construction costs compared with the text-retrieval baselines?
We now report token usage in Appendix D.2. During inference, token usage for retrieving@3 and retrieving@5 depends on the length of the retrieved protocols, while GARFIELD’s usage depends on the number of tool calls and the information returned by each call. Averaged across BioProBench and BioPIE, retrieving@3 uses 6.3k tokens per query, retrieving@5 uses 10k, and GARFIELD uses 9.1k. GARFIELD therefore achieves better performance than retrieving@5 while using fewer tokens per query. The average inference time is 5.8 s per query for retrieving@3, 7.6 s for retrieving@5, and 10.1 s for GARFIELD.
Token usage during one-time offline graph construction depends on protocol length. Constructing the Hierarchical, Execution, and Action-Context Views, including preprocessing and excluding entity normalization, uses an average of 20k input tokens per protocol, or approximately 25–30k total tokens including model outputs. Entity normalization for the Cross-Protocol Alignment View then uses approximately 75 tokens per local action or experimental object. Each batch call also includes approximately 360 instruction tokens, shared across the entities processed in that batch. We used Claude Opus-4.8 as the core model for all these measurements.
Manuscript: We added these measurements to Appendix D.2.
Thank you for the clarifications and additional cost measurements. However, comparisons with equivalent flat structured representations and graph-view ablations remain missing, leaving the contribution of the multi-view design insufficiently established. I therefore maintain my original score of 1 (clear rejection).
Response to Reviewer ivGM - Part1
We thank the reviewer for recognizing the motivation for GARFIELD as well as the breadth of evaluations. Following the raised concerns, we have extended the Methods section, providing a more self-contained framing in the main text, added an execution trace, and elaborated on token consumption details.
Following the reviewer’s comment, we have expanded the Methods section, and we hope you find it more self-contained. Specifically, we added:
Graph construction (Section 3.2): Added a dedicated subsection describing the complete pipeline, from protocol segmentation and hierarchical extraction to execution-flow construction, action context, cross-protocol alignment, and deterministic validation.
Agent and tool interface (Box 1: Tool functions and invocation format): We added descriptions of representative graph tools, their functions, and the JSON invocation format, clarifying how the agent discovers and uses tools.
Manuscript: Added Section 3.2 to describe the graph construction and Box 1 in Section 3.3 to describe the agent tool interface.
We appreciate this criticism. It is true that Figure 1 provides an abstract representation of the GARFIELD framework, highlighting its utility and overall organization. However, Figure 2 provides an example of open-ended protocol question answering, demonstrating tool invocation. We believe the core mechanism of GARFIELD is two-fold, comprising both the graph structure and abstraction and the tool-calling agent. We therefore devote figures to each aspect.
Thank you for making this suggestion. We added a complete illustrative execution trace of the GARFIELD agent for a BioProBench question, following the workflow in Algorithm 1, provided in Appendix G (Figure S10).
The trace shows the agent’s reasoning steps, tool calls, the graph relations accessed, returned results, and final answer. In this example, the tools search the step text, follow the NEXT and HAS_ACTION relations, and retrieve the parameters of the identified action.
Manuscript: Added Figure S10.