- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes SciCompiler, a system that transforms public domain-specific models into tools that agents can directly employ. It also involves assessing output consistency, interface costs, and the influence of tool documentation. The main contributions lie in the complete system organization and verifiable capability records. The paper conducts experiments on protein and genome tasks.
Critical strengths
- The paper focuses on a valuable problem: scientific agents need to reuse already developed domain-specific tools.
- Experimental results are reported with details of the intervention, including uncertainty about the cost of the DNA interface and the costs of acquisition.
Critical weaknesses
-
The technical contribution of this paper is limited. The SciCompiler could be viewed as an additional component for enhancing the Biomni, which utilizes Paper2Agent to transform the searched paper into reusable tools.
-
The method is not clear. The paper only describes the "capability decision" with several action names, without further information on how the SciCompiler evaluates the task. It is also not clear how these actions are selected, along with the assessment process.
-
For the "Scientific admission and retention," most of the task is processed by Paper2Agent and Biomni. It remains unclear what the significance of SciCompiler is in this workflow.
-
The discussion of the results is not informative. Most of the content describes the numbers instead of insights gained from the experiment. For example, how SciCompiler helps Biomni handle a specific error.
Questions for authors
-
Please provide the acquisition results for each task without intervention and with intervention.
-
Could you add tasks that are not suitable for reuse, implement the decisions induced by the tool card, and compare the final quality with the cumulative cost?
4 comments
Thank you for your responses. However, my concerns about the technical contribution, the significance of SciCompiler, the discussion of the results, and the clarity of the method still exist.
Could you provide more discussion on these issues in the weaknesses section?
Thank you for the feedback. SciCompiler adds a task-driven acquisition and verification workflow on top of tool compilation: it compiles an operation tailored to the task requirements, then checks execution evidence before accepting any output for scoring or reuse. Ligand-conditioned sequence design illustrates how this works in practice.
From task to action. Here, the input is an anonymous protein backbone plus ligand coordinates/elements, and the requested output is a sequence conditioned on both. Consider an inventory that starts with a backbone-only designer. Given the task requirements, the decision LLM inspects the capability records and selects one of three actions: reuse a compatible ligand-capable tool if listed, acquire and compile a suitable repository if missing, or decline if the requested evidence cannot be produced under current constraints (e.g., experimentally measured affinity when no measurement records exist). It outputs the chosen action, rationale, and supporting evidence; the controller validates the structured response and selected capability ID before dispatching downstream recommendation or execution runs with task-grounded arguments.
Added acquisition and execution checks. During acquisition, SciCompiler screens candidate repositories against the ligand requirement, locks the official source and checkpoint, and passes the required inputs and inference settings into compilation. Paper2Agent prepares the environment, extracts tools, and generates the MCP server; Biomni registers and invokes it. SciCompiler then checks that the actual run used the required context and settings and produced the submitted output. As described in §3.2, programmatic checks and independent trace auditing enforce this admission gate, binding each accepted tool to its implementation, checkpoint, runtime, and execution trace. Simply put, the manifest records what must hold, and execution admission checks it at runtime.
This directly changes acceptance outcomes. In the added LASErMPNN stress test, setting ignore_ligand=True still produces a correctly formatted sequence through MCP. Across two inputs, all 10 calls pass standard MCP and output-format checks, whereas SciCompiler's execution checker (§3.2) catches the mismatched inference argument: scientific admission accepts both normal controls and rejects all 8 injected faults (skipped inference, changed seed, disabled ligand conditioning, and extra inference calls). Its checks inspect actual inference arguments, parser context, seeds, call counts, checkpoint reads, and output provenance. Valid MCP registration and well-formed outputs are therefore insufficient on their own—the actual computation must be verified at execution time.
Key takeaways from the experiments. First, runtime verification is essential because output syntax alone cannot tell whether ligand conditioning was actually used during inference. Second, exposing verified operations through a convenient interface reduces agent overhead: in the paper's 20-structure comparison, our MCP interface preserves predictions while reducing invocation requests by 23.4% compared to direct Python (9.4→7.2 requests per task). Third, accurate capability descriptions stop unsupported requests before any scientific tool is invoked. In the measurement-only follow-up, both accurate-description conditions decline all six affinity-measurement requests up front without scientific calls—cleanly separating sequence prediction from experimental affinity measurement. Scope descriptions support coarse task filtering; for invoked tools, the manifest specifies the execution requirements and scientific admission verifies the actual run.
Thank you for your detailed responses. My concerns about the contribution of this manuscript still exist. It is more of a workflow to orchestrate the well-established Biomni and Paper2Agent. The significance of this workflow is limited. Therefore, I will maintain my current score, considering the contribution of this work.
Action-selection policy, intervention breakdown (Q1), and end-to-end card execution & cost (Q2)
Thank you for prompting us to tie action selection directly to execution. We have clarified the selection policy and traced tool-card decisions through to final scientific outputs and termination.
How actions are selected. Given the task and current inventory (each tool's inputs, scientific scope, and execution history), the LLM must either identify a tool and justify how its inputs and scope match the task, request new acquisition, or decline. The controller validates this structured response before dispatching an action. We will include the decision prompt in the Method section and explicitly separate standalone recommendation ablations from execution controllers.
Ligand conditioning illustrates how action selection connects to execution: in our new stress test, disabling conditioning still passes MCP interface and format checks, yet the existing execution checker automatically rejects the invalid inference configuration.
Q1. Results before and after intervention. The table summarizes all main-result acquisitions (recovery and accuracy in %).
Both protein tasks ran with zero manual intervention. The DNA interventions strictly addressed isolated sandbox configuration and submission-adapter parsing rather than the compiled tools themselves. For regulatory likelihood, restoring system loader paths allowed the already-compiled operation to run, and the evaluation adapter read the explicitly returned prediction JSON while ignoring auxiliary metadata paths. We applied this uniform submission rule throughout; every scientific prediction file remained byte-for-byte unchanged. The resulting capability was reused across regulatory, motif, and variant tasks. For accessibility, restoring Git metadata in the sandbox allowed all five compiler stages to pass; execution remained blocked solely because third-party licensed-weight credentials were absent from the offline sandbox (recorded in Appendix H).
Q2. Executing tool-card decisions, refusal on unsuitable tasks, and cumulative cost. We tested three tool cards: the original description, an accurate updated description, and the updated description with execution history. Following policy decisions into execution yielded 24 live scientific calls, all passing execution checks. All five paired sequence-design outputs matched identically in sequence and recovery score.
To isolate refusal behavior cleanly from valid sequence-design subtasks, we conducted a measurement-only follow-up requesting experimentally measured binding affinity. Both updated cards achieved the protocol minimum of 6 requests (one per task, with 6 valid declines and zero scientific calls). By contrast, the original card required 14 requests—producing only 3 declines, 1 fruitless search, 1 format failure, and 1 transport failure.
Cumulative cost. Reusing the pre-built tool across all control and follow-up runs added 62 generation requests (36 and 26, including search and failed attempts) with zero recompilation, bringing the total across original acquisition, first use (225 requests), and all follow-up ablations to 287. In a direct, matched comparison including compilation and first use, SciCompiler uses 205 requests versus 248 for Paper2Agent+Biomni. Post-compilation, the 20-structure study cuts requests by 23.4% (saving 2.2 requests and 34.5 worker seconds per task on average) with zero prediction degradation across all 20 matched pairs.