acceptodds
Under review as a conference paper at ICLR 2027

What Makes Agents Effective for Scientific Design? A Controlled Evaluation with Scientific Foundation Model Tools

Abstract

Scientific agents have shown promise for automating complex computational workflows, but their usefulness for scientific design hinge on reliably completing a task and finding high-quality solutions. Here, we evaluate how agent design choices—including the agentic harness, Large Language Model (LLM), and tool-calling mechanism—affect these two outcomes. We introduce Nomad, a platform for serving Scientific Foundation Models (SciFMs) through either direct tool calling or Programmatic Tool Calling (PTC) via the Model Context Protocol, allowing us to vary the tool-calling mechanism without modifying the tools or agentic harnesses. Our evaluation includes 845 agent rollouts across two multi-objective design tasks: designing a multi-material system for Richtmyer–Meshkov Instability formation and screening molecular space for liquid-electrolyte candidates. We measure design quality using Pareto hypervolume and fit a Bayesian hurdle model to separately estimate the probability of viable designs and the performance of the proposed designs conditional on success. We find that PTC most clearly improves reliability and performance for the multi-material design task, while the effects of harness and LLM are generally smaller and depend on the task and their pairing. Providing historical designs increases conditional hypervolume by 106% for the multi-material task, with agent traces suggesting that these examples help keep search near the surrogate model’s training distribution. Finally, higher-fidelity multi-physics simulations confirm that selected agent-proposed designs increased the historical Pareto-front hypervolume by +45% despite imperfect surrogate calibration. Collectively, our results demonstrate the importance of tool-calling mechanisms and scientific context when connecting SciFMs to agents for design tasks.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
AI Agents, Programmatic tool calling, Scientific foundation models, Multi-objective optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PY4qOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies which agent-design choices matter for scientific design. The authors introduce Nomad, which exposes the same scientific foundation/surrogate models through either direct tool calls or programmatic tool calling (PTC). Across 845 rollouts on continuous perturbed-layer-interface (PLI) design and discrete electrolyte screening, the study varies agent harness, backing LLM, tool-calling mechanism, and—on PLI—access to historical designs. Performance is summarized by Pareto hypervolume and analyzed with a Bayesian hurdle model that separates the probability of producing a viable design from quality conditional on success. PTC is estimated to increase pooled reliability by 1.5× and conditional hypervolume by 42%; historical PLI designs increase conditional hypervolume by 106.4%. PAGOSA simulations of selected PLI candidates show an agent-proposed Pareto front with hypervolume 8.0 versus 5.53 for historical designs, despite imperfect surrogate calibration. The electrolyte study remains surrogate-only.

Critical strengths
  1. This is a systematic evaluation of scientific agents. The study spans multiple harnesses, LLMs, tool interfaces, two qualitatively different design spaces, and 845 replicated rollouts. Separating task reliability from conditional design quality with a hurdle model is appropriate and substantially more informative than reporting task completion alone.

  2. The higher-fidelity PLI validation is a major strength. The authors explicitly measure JANUS–PAGOSA calibration and show that selected designs retain improvement under expensive simulation. The extensive diagnostics, released traces/code, and careful caveats about surrogate-only molecular results and unequal optimizer comparisons improve the work’s transparency.

Critical weaknesses
  1. The main PTC effect is confounded with effective search budget. PTC is specifically able to batch many surrogate evaluations and use numerical libraries within one invocation, whereas direct tool calls expose each result through the conversation. Rollouts share a four-hour wall-time limit, but apparently not a matched number of surrogate evaluations, candidates considered, tool calls, or tokens. Pareto hypervolume is monotone in the size of the candidate set, so a condition that can evaluate/report more candidates has a structural advantage even without better scientific reasoning. Although tool, token, and wall-time usage were logged, the main analysis does not normalize or condition on these quantities. This makes it unclear whether PTC improves search quality per evaluation or mainly enables more search.

  2. The scope and comparator evidence limit the broad conclusions. The cost-aware subset design is unbalanced and some model–harness cells may be sparse, while only two tasks are studied. The PAGOSA comparison also uses unequal search and validation budgets: 84 agent designs versus 15 NSGA-II designs are simulated, and resampling 15 agent designs gives hypervolume 6.93 (95% HDI 5.85–7.81), below the reported NSGA-II value of 7.95. The paper appropriately notes that it does not establish superiority, but “competitive” remains sensitive to this asymmetry. The proprietary PAGOSA implementation further limits independent reproduction of the key high-fidelity result.

Questions for authors
  1. Does the PTC effect persist when direct and PTC agents are matched on surrogate-evaluation count and when each rollout is scored from the same number of randomly subsampled candidates? Please report hypervolume versus evaluations, tokens, and wall time. This is the most important issue for my assessment.

  2. How many infrastructure/external-service runs were excluded in each treatment group? Are the hurdle-model conclusions robust to treating these exclusions as failures or to inverse-probability/sensitivity analyses?

  3. Under equal surrogate and PAGOSA validation budgets, how do agents compare with NSGA-II? In particular, can the authors report repeated equal-size subsampling for both fronts and uncertainty for their hypervolume difference?

1 comment

Thank you for identifying the distinction between overall rollout performance and performance at a common search budget. We have added analyses addressing search budgets, exclusions, and validation.

Q1. Our primary analysis estimates the overall effect of the tool interface under a common four-hour limit. We omit realized resource usage from the primary hurdle model because it is a post-treatment variable. To address the reviewer's efficiency question, Fig. 7 shows hypervolume against evaluation count, token usage, and elapsed wall time for PLI design and electrolyte screening. Table 8 uses the first N evaluations of each trajectory that reached the indicated count and reports hypervolume conditional on agent success. Table 6 gives the counts of eligible runs with both zero and positive hypervolume, alongside the exclusions. We interpret the fixed-count comparisons descriptively because the mix of configurations varies with the evaluation count.

Adding realized resource usage to the hurdle model improved predictive fit primarily for rollout success and added little to our ability to predict final hypervolume among successful runs. Table 8 also shows how far individual PLI searches extended. The largest direct-calling run evaluated 136 designs, compared with 2,191 for PTC. This difference is consistent with greater search volume contributing to the overall PTC advantage.

Q2. Table 6 reports exclusions by task, LLM, harness, tool interface, and data access. We excluded 153 of 774 PLI runs and 10 of 234 electrolyte runs. As a descriptive sensitivity check, counting every excluded run as unsuccessful gives PLI success rates of 39.7% for PTC (196/494) and 23.6% for direct calling (66/280). For electrolyte screening, the rates are 44.0% (59/134) and 23.0% (23/100), respectively. The direction of the unadjusted comparison persists in both tasks. We have not refit the adjusted hurdle model under this alternative rule.

Q3. We allocated 100 PAGOSA simulations to agent designs and 100 to NSGA-II designs, doubling the original validation cohort. Simulations completed for 99 agent designs and all 100 NSGA-II designs. The measured hypervolumes were 8.07 and 8.55, respectively, compared with 5.53 for the historical front (Fig. 3). The revised Methods describe how we selected designs from each group.

To compare search performance at a common evaluation count, we used the expanded validation data to calibrate JANUS predictions against PAGOSA. At 500 evaluations, median predicted hypervolume was 8.22 for the five successful PTC runs with historical data that reached that count (95% HDI 7.35–9.17) and 8.43 for ten NSGA-II runs (7.53–9.52). Figure 7 and Table 8 show how the comparison varies with evaluation count.

Reviewer_aZwdOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper conducts a bunch of experiments measuring the performance of agents on performing two science tasks, scanning a grid of 3 parameters (which harness, how tool calls are handled, which LLM). The paper also checks, for one of the tasks, if historical data is useful (unsurprisingly, it is).

The question in the title: "WHAT MAKES AGENTS EFFECTIVE FOR SCIENTIFIC DESIGN?" is not answered by the paper, whose conclusion is that the parameters in the grid search did not result in meaningful differences. One difference that was found is that letting the agent write code for tool use is better than only giving it tool calling access via MCP, but this is well documented by now in many papers.

It is not clear to me what is the takeaway I should get from this paper that is transferable to other scenarios (even to other scientific agent scenarios), that was not already well known.

Critical strengths

None.

Critical weaknesses

The work does some experiments and report their results, but does not meaningfully answer any interesting research question, and does not sufficiently contribute to our knowledge. It also does not answer the question posed in its own title.

Questions for authors

None.

1 comment

We agree that prior work established the value of using code to coordinate tool calls. We have revised the Introduction and Conclusion to make our empirical question and results clearer.

On both tasks, the estimated effects of PTC on reliability and design quality were positive, although both effects were less certain for molecular screening. On PLI, historical context produced the largest estimated gain in design quality among the treatments we evaluated.

Distinguishing the probability of success from the quality of a successful result matters because a configuration change can affect the two outcomes differently. On PLI, providing historical data roughly doubled expected hypervolume among successful runs, while its effect on the probability of success was uncertain. A benchmark that scored only task completion would have missed the largest estimated gain in design quality in our PLI experiments.

Reviewer_wGLxOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to configure agents for scientific design using existing prediction models as tools. It introduces Nomad to connect these tools to different agent frameworks. Agents can call tools directly or write Python programs that coordinate evaluations and process the results.

The authors conduct 845 runs on two tasks: optimizing a multi-material interface and screening molecules for electrolyte-solvent properties. They compare language models, agent frameworks, tool-calling methods, and access to historical designs. The evaluation separately measures how often a run produces a design set with positive hypervolume and how good that set is when successful.

On the interface-design task, programmatic tool calling improves both the probability of success and the quality of successful outputs. Providing historical designs roughly doubles the expected quality conditional on success. Selected candidates are then evaluated with a higher-fidelity simulator, where the agent-produced design set improves on the historical set and achieves a hypervolume close to the NSGA-II result.

Critical strengths
  1. The study separately measures how often agents produce useful designs and how good those designs are when successful. At least five repeat runs were launched per configuration. Candidates are reevaluated with the scientific prediction tools, with ten JANUS predictions averaged per PLI design. This gives the configuration comparisons a clear evaluation procedure.

  2. Higher-fidelity simulation confirms that the selected agent designs improve on the historical PLI design set. This connects the search results to a more accurate physical evaluation. The NSGA-II experiments also provide a conventional optimization reference for assessing the quality of the designs.

Critical weaknesses
  1. The study adds limited insight beyond established practices. Code-based tool coordination is already supported by work such as CodeAct. The PLI experiments confirm a benefit, but do not explain which features of programmatic calling matter or when the benefit should transfer. Historical data also helps, while narrowing search toward meaningful designs. These results provide application evidence without substantially deepening our understanding of agent configuration.

  2. The conclusion calls model and framework effects minimal, despite substantial differences in some settings. For example, Table 4 estimates Sonnet's PLI success probability at about half that of GPT-5.4 when other factors are fixed at the reference settings. No consistently best combination does not mean these choices have little effect. The practical advice supported by the comparisons needs to be clearer.

Questions for authors
  1. Which finding most changes our understanding beyond existing work on code-based tool use? Can the recorded traces or measurements explain why PTC helps on PLI and whether the same explanation applies to molecular screening? This would help clarify what general lesson the experiments add.

  2. What specific advice about model and framework choice follows from the reported results? Please explain how the conclusion of minimal effects fits the substantial conditional differences and the task-dependent results in Table 4.

1 comment

Q1. On PLI, PTC increased both the probability of returning a viable design (1.8x) and the expected hypervolume of successful runs (+22%). A completion-only evaluation, as explored in prior work on PTC, would have missed the second effect.

For PLI, PTC enabled agents to explore a much larger design space. As shown in Table 8, agents using PTC evaluated up to 2,191 designs, compared to 136 for direct tool calling. This difference is consistent with greater search volume contributing to the overall PTC advantage. From Fig. 7, we also see that, at the same evaluation budget, both paradigms produced similar hypervolumes.

For molecular screening, the largest runs evaluated 136 molecules with PTC and 64 with direct tool calling, and the estimated effect of PTC on quality was uncertain (Table 4). The traces therefore provide a plausible explanation for the PTC uplift on the PLI design task, with weaker evidence for the same explanation in molecular screening.

Q2. We agree that “minimal” understated the conditional differences in Fig. 2 and Table 4. At the reference settings, Sonnet was about half as likely as GPT-5.4 to return a viable PLI design (relative risk 0.51, 95% HDI 0.28–0.77). Harness choice could also affect quality among successful runs even when its effect on reliability was uncertain. We revised the Results and Conclusion to give these comparisons explicitly. For a new task, our advice is to evaluate both outcomes for the model–harness pairing under consideration and weigh the cost of a failed run against the value of a better result.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.