acceptodds
Under review as a conference paper at ICLR 2027

SciCompiler: Compiling Open Scientific Artifacts into Verified Agent Capabilities

Abstract

Scientific advances are often released as papers, repositories, and checkpoints rather than as capabilities that scientific agents can directly use and retain, while existing agents still rely largely on fixed tool repertoires. We introduce SciCompiler, which discovers artifacts for a given task and compiles them from locked source into callable operations, independent execution validation, and capability registration. Each retained capability binds a scientific operation to its interface, provenance, source, weights, runtime, and execution evidence, allowing subsequent tasks to invoke the same validated computation without repeating acquisition. Across protein design and genomics, SciCompiler acquires reusable scientific operations that remain callable on new inputs and across downstream workflows. In genomics, a retained Evo 2 likelihood capability reaches 96.25% regulatory accuracy on reserved chromosomes, compared with 78.59% for the coding baseline, and the same operation is reused across regulatory, motif, and variant tasks. Controlled studies further show that structured MCP invocation preserves scientific outputs while reducing agent requests by 23.4% and worker time by 22.3% in protein tasks. Augmenting capability records with artifact identity and validated execution facts increases reuse recommendations by 20 percentage points. Together, these results show how scientific agents can grow beyond fixed tool inventories by turning research artifacts into validated, reusable capabilities with explicit provenance and execution evidence.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Scientific Agents, LLM Agents, Tool Use, Open-World Tool Acquisition, AI for Science, Scientific Workflow Automation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dgeGOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes SciCompiler, a system that transforms public domain-specific models into tools that agents can directly employ. It also involves assessing output consistency, interface costs, and the influence of tool documentation. The main contributions lie in the complete system organization and verifiable capability records. The paper conducts experiments on protein and genome tasks.

Critical strengths
  • The paper focuses on a valuable problem: scientific agents need to reuse already developed domain-specific tools.
  • Experimental results are reported with details of the intervention, including uncertainty about the cost of the DNA interface and the costs of acquisition.
Critical weaknesses
  • The technical contribution of this paper is limited. The SciCompiler could be viewed as an additional component for enhancing the Biomni, which utilizes Paper2Agent to transform the searched paper into reusable tools.

  • The method is not clear. The paper only describes the "capability decision" with several action names, without further information on how the SciCompiler evaluates the task. It is also not clear how these actions are selected, along with the assessment process.

  • For the "Scientific admission and retention," most of the task is processed by Paper2Agent and Biomni. It remains unclear what the significance of SciCompiler is in this workflow.

  • The discussion of the results is not informative. Most of the content describes the numbers instead of insights gained from the experiment. For example, how SciCompiler helps Biomni handle a specific error.

Questions for authors
  1. Please provide the acquisition results for each task without intervention and with intervention.

  2. Could you add tasks that are not suitable for reuse, implement the decisions induced by the tool card, and compare the final quality with the cumulative cost?

4 comments
AuthorsOpenReview ↗

Action-selection policy, intervention breakdown (Q1), and end-to-end card execution & cost (Q2)

Thank you for prompting us to tie action selection directly to execution. We have clarified the selection policy and traced tool-card decisions through to final scientific outputs and termination.

How actions are selected. Given the task and current inventory (each tool's inputs, scientific scope, and execution history), the LLM must either identify a tool and justify how its inputs and scope match the task, request new acquisition, or decline. The controller validates this structured response before dispatching an action. We will include the decision prompt in the Method section and explicitly separate standalone recommendation ablations from execution controllers.

Ligand conditioning illustrates how action selection connects to execution: in our new stress test, disabling conditioning still passes MCP interface and format checks, yet the existing execution checker automatically rejects the invalid inference configuration.

Q1. Results before and after intervention. The table summarizes all main-result acquisitions (recovery and accuracy in %).

AcquisitionOriginal resultResult after environment alignment
Protein design, ADFLIP52.80 whole-chain recoveryNo manual intervention
Protein design, LASErMPNN57.98 whole-chain recovery (via built-in auto package repair)No manual intervention
DNA regulatory likelihoodCompiled; eval sandbox omitted system-library pathsPaths restored: 95.31 initial accuracy; 96.25 on 5 reserved batches
DNA accessibilityEval checkout lacked required Git metadataAll 5 compiler stages passed; execution gated only by missing licensed-weight credentials in sandbox

Both protein tasks ran with zero manual intervention. The DNA interventions strictly addressed isolated sandbox configuration and submission-adapter parsing rather than the compiled tools themselves. For regulatory likelihood, restoring system loader paths allowed the already-compiled operation to run, and the evaluation adapter read the explicitly returned prediction JSON while ignoring auxiliary metadata paths. We applied this uniform submission rule throughout; every scientific prediction file remained byte-for-byte unchanged. The resulting capability was reused across regulatory, motif, and variant tasks. For accessibility, restoring Git metadata in the sandbox allowed all five compiler stages to pass; execution remained blocked solely because third-party licensed-weight credentials were absent from the offline sandbox (recorded in Appendix H).

Q2. Executing tool-card decisions, refusal on unsuitable tasks, and cumulative cost. We tested three tool cards: the original description, an accurate updated description, and the updated description with execution history. Following policy decisions into execution yielded 24 live scientific calls, all passing execution checks. All five paired sequence-design outputs matched identically in sequence and recovery score.

To isolate refusal behavior cleanly from valid sequence-design subtasks, we conducted a measurement-only follow-up requesting experimentally measured binding affinity. Both updated cards achieved the protocol minimum of 6 requests (one per task, with 6 valid declines and zero scientific calls). By contrast, the original card required 14 requests—producing only 3 declines, 1 fruitless search, 1 format failure, and 1 transport failure.

Cumulative cost. Reusing the pre-built tool across all control and follow-up runs added 62 generation requests (36 and 26, including search and failed attempts) with zero recompilation, bringing the total across original acquisition, first use (225 requests), and all follow-up ablations to 287. In a direct, matched comparison including compilation and first use, SciCompiler uses 205 requests versus 248 for Paper2Agent+Biomni. Post-compilation, the 20-structure study cuts requests by 23.4% (saving 2.2 requests and 34.5 worker seconds per task on average) with zero prediction degradation across all 20 matched pairs.

Thank you for your responses. However, my concerns about the technical contribution, the significance of SciCompiler, the discussion of the results, and the clarity of the method still exist.

Could you provide more discussion on these issues in the weaknesses section?

Thank you for the feedback. SciCompiler adds a task-driven acquisition and verification workflow on top of tool compilation: it compiles an operation tailored to the task requirements, then checks execution evidence before accepting any output for scoring or reuse. Ligand-conditioned sequence design illustrates how this works in practice.

From task to action. Here, the input is an anonymous protein backbone plus ligand coordinates/elements, and the requested output is a sequence conditioned on both. Consider an inventory that starts with a backbone-only designer. Given the task requirements, the decision LLM inspects the capability records and selects one of three actions: reuse a compatible ligand-capable tool if listed, acquire and compile a suitable repository if missing, or decline if the requested evidence cannot be produced under current constraints (e.g., experimentally measured affinity when no measurement records exist). It outputs the chosen action, rationale, and supporting evidence; the controller validates the structured response and selected capability ID before dispatching downstream recommendation or execution runs with task-grounded arguments.

Added acquisition and execution checks. During acquisition, SciCompiler screens candidate repositories against the ligand requirement, locks the official source and checkpoint, and passes the required inputs and inference settings into compilation. Paper2Agent prepares the environment, extracts tools, and generates the MCP server; Biomni registers and invokes it. SciCompiler then checks that the actual run used the required context and settings and produced the submitted output. As described in §3.2, programmatic checks and independent trace auditing enforce this admission gate, binding each accepted tool to its implementation, checkpoint, runtime, and execution trace. Simply put, the manifest records what must hold, and execution admission checks it at runtime.

This directly changes acceptance outcomes. In the added LASErMPNN stress test, setting ignore_ligand=True still produces a correctly formatted sequence through MCP. Across two inputs, all 10 calls pass standard MCP and output-format checks, whereas SciCompiler's execution checker (§3.2) catches the mismatched inference argument: scientific admission accepts both normal controls and rejects all 8 injected faults (skipped inference, changed seed, disabled ligand conditioning, and extra inference calls). Its checks inspect actual inference arguments, parser context, seeds, call counts, checkpoint reads, and output provenance. Valid MCP registration and well-formed outputs are therefore insufficient on their own—the actual computation must be verified at execution time.

Key takeaways from the experiments. First, runtime verification is essential because output syntax alone cannot tell whether ligand conditioning was actually used during inference. Second, exposing verified operations through a convenient interface reduces agent overhead: in the paper's 20-structure comparison, our MCP interface preserves predictions while reducing invocation requests by 23.4% compared to direct Python (9.4→7.2 requests per task). Third, accurate capability descriptions stop unsupported requests before any scientific tool is invoked. In the measurement-only follow-up, both accurate-description conditions decline all six affinity-measurement requests up front without scientific calls—cleanly separating sequence prediction from experimental affinity measurement. Scope descriptions support coarse task filtering; for invoked tools, the manifest specifies the execution requirements and scientific admission verifies the actual run.

Thank you for your detailed responses. My concerns about the contribution of this manuscript still exist. It is more of a workflow to orchestrate the well-established Biomni and Paper2Agent. The significance of this workflow is limited. Therefore, I will maintain my current score, considering the contribution of this work.

Reviewer_WCXDOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

SciCompiler proposes a framework for expanding scientific agents’ capabilities by converting research artifacts into callable and reusable operations. Given a task and an existing capability inventory, the system decides whether to reuse a capability, acquire a new one, or decline the task. Acquisition combines artifact discovery, source locking, compilation through Paper2Agent, package validation, and a separate scientific admission process. Accepted capabilities retain their interface, source, weights, runtime, and execution evidence.

The evaluation covers protein design and genomics, alongside controlled studies of invocation interfaces and capability records. The central claim is that scientific capabilities should preserve both executable computation and the evidence needed to recognize and reliably reuse it.

Critical strengths

Strengths

  • The execution protocol goes beyond checking whether a tool returns plausible outputs. The paper explicitly distinguishes package-level tests and successful MCP registration from evidence that the intended scientific computation actually occurred. Its admission process considers source and checkpoint access, task-specific settings, relevant input pathways, and the validation of submitted predictions. This is valuable because a technically functional wrapper can still omit scientifically necessary inputs or invoke an unintended computation.

  • The protein interface study provides a reasonably controlled empirical result. Direct Python and MCP invocation use the same retained function, weights, runtime, inputs, and inference settings. All twenty pairs produce identical sequences, while MCP reduces mean requests by 23.4% and native execution time by 22.3%. Reporting paired intervals, execution-order strata, and the substantially smaller median time difference helps characterize both the result and its limitations.

  • The reporting acknowledges important qualifications. The manuscript discloses failed attempts, manual interventions, incomplete outcomes, and the substantial initial acquisition investment. It also distinguishes capability-selection recommendations from executed actions and reports the DNA interface replication despite its inconclusive cost differences. These details make the empirical claims more interpretable.

Critical weaknesses

Weaknesses

  • The method's contribution relative to existing systems is not sufficiently isolated through ablations. Paper2Agent (Miao et al., 2026) already provides research-artifact compilation into tested MCP tools; SciToolAgent-Evo (Tang et al., 2026) studies capability gaps and tool acquisition; SkillFoundry (Shen et al., 2026) constructs validated reusable scientific skills; and SciAgent (Badhwar, 2026) and ScienceIDE (Geng et al., 2026) address execution integrity or scientific validation. SciCompiler combines these directions into a coherent workflow, but the paper needs a more direct and straightforward explanation (ideally a set of experiments) of which additional mechanism constitutes its main advancement. The current component studies do not establish the incremental benefit of the complete acquisition-and-admission pipeline over simpler combinations of existing approaches.

  • The main DNA accuracy improvement cannot be attributed specifically to the framework. The reported 96.25% versus 78.59% comparison evaluates Evo 2 7B against Nucleotide Transformer 500M, with different model families and scales. Moreover, the reported Evo 2 configuration depends on a compatibility continuation and a submission-envelope intervention. These results therefore provide weaker support for the paper's central claim than the accuracy comparison might suggest. Maybe ablation within the methods, i.e., about different components, will solve this?

  • The scientific admission process remains partly dependent on author judgment. The manuscript states that admission combines programmatic checks with author-supervised inspection of process and file traces, but does not clearly enumerate which criteria are automatically enforced and which require manual examination, and how these manual checks are performed. A more explicit specification of the checks, together with controlled failure-injection experiments, would help assess whether this component is a reusable verification mechanism or primarily an experimental auditing procedure.

Questions for authors

As mostly stated in the weaknesses,

  • What is the essential mechanism added by SciCompiler beyond combining Paper2Agent with artifact search, capability registration, and execution logging? Can an ablation demonstrate the benefit of this mechanism over such a simpler pipeline?

  • Which criteria are enforced with hard-coded programs, and which require author inspection? Have you measured whether admission detects deliberately introduced failures?

  • How can the DNA improvement be separated from the difference between Evo 2 7B and Nucleotide Transformer 500M? Would the advantage persist when both routes receive the same candidate artifacts or when acquisition is repeated across additional independent tasks?

3 comments
AuthorsOpenReview ↗

Execution stress test, provenance clarification, and fixed-predictor control

Thank you for the concrete suggestions. We have added an execution stress test and a fixed-predictor control to isolate workflow overhead from model choice.

What SciCompiler adds. Consider ligand-conditioned sequence design: a wrapper can emit a syntactically valid amino-acid sequence even while silently dropping the ligand. SciCompiler propagates task-level scientific constraints into compilation, verifies that the executed run satisfies them, and caches the validated tool for reuse. While Paper2Agent builds tools and Biomni invokes them, SciCompiler bridges the two by enforcing task requirements—such as loading the right checkpoint and ensuring ligand conditioning reaches inference.

Execution-check fault coverage. We added a deterministic fault-injection stress test covering four representative execution errors: skipping inference, altering the random seed, disabling ligand conditioning, and running an extra inference pass. The test runs 10 live MCP calls across two existing inputs using LASErMPNN with unchanged scientific source code and weights, evaluated by the checker already in our submitted codebase.

Test groupCallsMCP and output-format checksExecution checker
Normal controls2PassedAccepted
Four injected fault types8PassedRejected

The checker caught all eight injected faults while accepting both normal controls; standard MCP schema and output-format checks missed every single fault.

Specifically, the automatic checker inspects seeds, inference arguments, call counts, parser logs, checkpoint reads, and submitted files. For DNA benchmarks, §3.2 additionally includes a pre-scoring provenance audit of completed-run logs to confirm which tool produced the output file. This audit checks execution provenance rather than task scores and never generates or edits predictions.

Fixed-predictor comparison. In a new compilation-and-first-use control, SciCompiler and task-scoped Paper2Agent+Biomni receive the exact same artifact, weights, task, settings, tutorials, and budgets. Both produce identical sequences and scores (70.00% whole-chain recovery; 63.79% interface recovery), while SciCompiler cuts total requests—including compilation—from 248 to 205 (17.3%).

In the 20-structure study with the model held fixed, the compiled interface reduces requests by 23.4%, worker time by 22.3%, and prompt tokens by 70.5%, with exact prediction agreement across all 20 pairs. When task accuracy is bottlenecked by the model itself, acquiring a stronger predictor yields substantial gains: the acquired Evo 2 capability reaches 96.25% versus 78.59% accuracy on the five reserved regulatory batches (+17.66 points, 95% CI [14.38, 21.09]).

Reviewer_WCXDOpenReview ↗

Reply to author's rebuttal

Thanks for your rebuttal. The additional numbers are helpful, but they do not yet convince me that SciCompiler provides a sufficiently advantage over other methods. To better assess whether they address my concerns, could you clarify the following?

  • Performance diff mechanism: In the new fixed-predictor comparison, what exactly differs between SciCompiler and task-scoped Paper2Agent+Biomni in their pipeline? Which additional component accounts for the request reduction?

  • Underlying agent model: Do both systems use the same underlying base LLM? If so, which? The rebuttal specifies the framework but didn't provide any further details, preventing me from distinguishing whether the gain rooted from better framework or a performance gap in the underlying models.

  • Exact numbers: you mentioned receive the exact same artifact, weights, task, settings, tutorials, and budgets. Can you elaborate detailed settings, especially the number of cases? Are you using the same setting as one of the experiments in the paper? If not, why?

Thank you for raising this control question. We added a controlled comparison that fixes both the underlying predictor and the tutorial scope to measure the total requests needed to compile a tool and obtain the exact same prediction. SciCompiler reduces total requests from 248 to 205 (-17.3%) across compilation and first use, with exact sequence agreement.

Pipeline difference and mechanism. Both setups run Paper2Agent's five compilation stages from scratch and use the same native Biomni A1 agent. SciCompiler supplies a task-specific I/O specification, a task-level wrapper interface, and a capability manifest that records artifact provenance. In contrast, the comparator relies on Paper2Agent's default generated schemas and logs. Both setups go through the same package checks and external execution review.

Concretely, SciCompiler wraps the model into a task-level interface, design_sequences(input_directory, output_directory, seed), which reads all target rows, resets the seed per row, and writes the predictions JSON directly. The comparator instead exposes a single-structure design_inverse_folding tool alongside separate parsing and loading helpers. During its recorded first use, A1 has to invoke those helpers manually, recover from MCP-result decoding issues, and assemble the final JSON from intermediate files. Moving multi-row iteration and output assembly into the wrapper eliminates this multi-step orchestration from the agent.

Generation requestsSciCompilerComparator
Environment/tutorial scan5968
Tutorial execution4934
Tool extraction5376
MCP wrapping1121
Coverage/quality reporting1219
Compilation subtotal184218
First use2130
Total205248

Overall, SciCompiler saves 43 requests (17.3%): 34 fewer requests during compilation and 9 fewer during first use. The largest savings occur in tool extraction (76→53) and MCP wrapping (21→11), where the prescribed interface constrains what the compiler needs to generate.

Underlying LLM. Both setups use the same API service with matched per-role configurations: the compiler uses adaptive thinking (effort=max), and A1 uses temperature 0.7 (default reasoning effort) with a 16,384-token output cap. API response logs confirm identical model identifiers across both runs (glm-5 for the compiler and glm-5.3 for A1).

Evaluation settings. We run a fresh compilation pass per setup on the benchmark structure W from Appendix C.1, evaluated across three input rows (context-present, context-removed, and an exact repeat). Both setups use the exact same locked ADFLIP source, ADFLIP_v1.pt, EMA, adaptive sampling (8 steps, threshold 0.9, argmax final), min_t=0, sidechain packing, temperature 0.1, noise 1, one CPU thread, per-row seed 37, and one sample per row. Task scope, tutorial-scoping policy, and budgets match (compilation: 600 requests / 9,000 s; first use: 40 requests / 900 s), with evaluation coordinates and references withheld during compilation.

While Appendix C.1 originally evaluated W with LASErMPNN under 620/20 compile/use request ceilings, fixing ADFLIP and aligning tutorial scope here directly verifies that workflow gains persist when the predictor and tutorials are held constant. The paper's 20-structure study separately evaluates repeated invocation after compilation.

Reviewer_Yxa8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors present SciCompiler, a methodology for expanding and storing validated agent tools and artifacts targeted for scientific tasks. The primary concept in the manuscript is that the selection and storage of relevant artifacts and validated for later iterations and use can reduce the intialization burden on later agent tasks.

Critical strengths

SciCompiler reaches a higher accuracy and lower cost than the provided benchmark on some scientific tasks.

Critical weaknesses
  1. The language of the manuscript makes a comprehensive review challenging as it is difficult to interpret the exact algorithm and experimental design that was implemented.

  2. The contributions of SciCompiler are not clear relative to prior literature on tooling and agent memory.

  3. The current benchmarks are insufficient for demonstrating the value of the proposed methods. The manuscript should include more competitive, state of the art methods.

  4. While the design could have relevant applications as an engineered system, the work does not present a clear concept or finding that significantly improves current systems or informs further development.

Questions for authors
  1. How does SciCompiler differ from existing tool use memory methods, including more generalized agent approaches?

  2. How does this method improve validated tool use over an MCP server that has more robust tools? Is this concept different from allowing an agent to revise its own tools?

2 comments
AuthorsOpenReview ↗

Positioning vs. tool memory & MCP servers, stress test, and fixed- vs. acquired-model results

Thank you for pushing us to clarify SciCompiler's positioning and experimental attribution. SciCompiler bridges external model discovery and reliable execution across downstream tasks, and we will sharpen both this distinction and the supporting results in the revision.

SciCompiler bridges external model discovery and reliable execution across downstream tasks. We will sharpen both this positioning and the supporting results in the revision.

Relationship to tool memory. In ligand-conditioned sequence design, a reusable capability must bind together weights, callable code, required structure/ligand inputs, and verified execution settings. SciCompiler acquires and validates this runnable package so that any memory module can safely retrieve it later.

Passing format checks alone does not guarantee valid science. In our new stress test, we injected four faults (disabling ligand conditioning, altering seeds, skipping inference, and adding an extra inference pass). All eight faulty runs returned valid MCP outputs, yet our existing execution checker caught all eight while accepting both normal controls. Reuse should therefore be gated on execution evidence, not merely endpoint availability.

Relationship to a robust MCP server and tool revision. SciCompiler can wrap a robust MCP server as its execution endpoint, adding task-directed model discovery, compilation, and verification that intended scientific constraints actually reach inference. Validated tools are reused as-is, while missing capabilities are compiled from released repositories. Repairs only modify generated wrappers and packaging while holding scientific source code and weights fixed—allowing us to compare invocation routes without altering the underlying calculation.

Fixed-model efficiency and acquired-model quality. In the paired 1gx8 benchmark (Appendix C), SciCompiler matches a coding agent's whole-chain recovery (46.58%) while doubling interface recovery from 26.67% to 53.33% over native Biomni and upstream Paper2Agent+Biomni.

In our new fixed-artifact control against Paper2Agent+Biomni (identical artifact, weights, task, settings, and budget), both achieve identical outputs while SciCompiler cuts compilation-and-first-use requests from 248 to 205.

In the 20-structure benchmark, invoking the same compiled function, weights, and settings via MCP instead of direct Python reduces mean requests by 23.4% (−2.20, 95% CI [−3.50, −1.00]), worker time by 22.3%, and prompt tokens by 70.5%, with zero prediction degradation across all 20 pairs. Finally, when external acquisition unlocks a stronger predictor, task accuracy improves substantially: the acquired Evo 2 tool achieves 96.25% versus 78.59% accuracy on five reserved regulatory batches (+17.66 points, 95% CI [14.38, 21.09]).

Thanks for the detailed response and the extra experiments. The fault-injection test and fixed-artifact control do help show the engineering utility of your execution checks. However, my main concerns still stand. The rebuttal actually confirms that SciCompiler is mainly an engineering wrapper and orchestration workflow built around Paper2Agent and Biomni, rather than offering a clear conceptual advance over existing agent tool-memory or learning frameworks. The lack of comparisons against broader SOTA agent baselines and the manuscript's clarity issues also remain unaddressed. Given these points, I will maintain my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.