- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies which agent-design choices matter for scientific design. The authors introduce Nomad, which exposes the same scientific foundation/surrogate models through either direct tool calls or programmatic tool calling (PTC). Across 845 rollouts on continuous perturbed-layer-interface (PLI) design and discrete electrolyte screening, the study varies agent harness, backing LLM, tool-calling mechanism, and—on PLI—access to historical designs. Performance is summarized by Pareto hypervolume and analyzed with a Bayesian hurdle model that separates the probability of producing a viable design from quality conditional on success. PTC is estimated to increase pooled reliability by 1.5× and conditional hypervolume by 42%; historical PLI designs increase conditional hypervolume by 106.4%. PAGOSA simulations of selected PLI candidates show an agent-proposed Pareto front with hypervolume 8.0 versus 5.53 for historical designs, despite imperfect surrogate calibration. The electrolyte study remains surrogate-only.
Critical strengths
-
This is a systematic evaluation of scientific agents. The study spans multiple harnesses, LLMs, tool interfaces, two qualitatively different design spaces, and 845 replicated rollouts. Separating task reliability from conditional design quality with a hurdle model is appropriate and substantially more informative than reporting task completion alone.
-
The higher-fidelity PLI validation is a major strength. The authors explicitly measure JANUS–PAGOSA calibration and show that selected designs retain improvement under expensive simulation. The extensive diagnostics, released traces/code, and careful caveats about surrogate-only molecular results and unequal optimizer comparisons improve the work’s transparency.
Critical weaknesses
-
The main PTC effect is confounded with effective search budget. PTC is specifically able to batch many surrogate evaluations and use numerical libraries within one invocation, whereas direct tool calls expose each result through the conversation. Rollouts share a four-hour wall-time limit, but apparently not a matched number of surrogate evaluations, candidates considered, tool calls, or tokens. Pareto hypervolume is monotone in the size of the candidate set, so a condition that can evaluate/report more candidates has a structural advantage even without better scientific reasoning. Although tool, token, and wall-time usage were logged, the main analysis does not normalize or condition on these quantities. This makes it unclear whether PTC improves search quality per evaluation or mainly enables more search.
-
The scope and comparator evidence limit the broad conclusions. The cost-aware subset design is unbalanced and some model–harness cells may be sparse, while only two tasks are studied. The PAGOSA comparison also uses unequal search and validation budgets: 84 agent designs versus 15 NSGA-II designs are simulated, and resampling 15 agent designs gives hypervolume 6.93 (95% HDI 5.85–7.81), below the reported NSGA-II value of 7.95. The paper appropriately notes that it does not establish superiority, but “competitive” remains sensitive to this asymmetry. The proprietary PAGOSA implementation further limits independent reproduction of the key high-fidelity result.
Questions for authors
-
Does the PTC effect persist when direct and PTC agents are matched on surrogate-evaluation count and when each rollout is scored from the same number of randomly subsampled candidates? Please report hypervolume versus evaluations, tokens, and wall time. This is the most important issue for my assessment.
-
How many infrastructure/external-service runs were excluded in each treatment group? Are the hurdle-model conclusions robust to treating these exclusions as failures or to inverse-probability/sensitivity analyses?
-
Under equal surrogate and PAGOSA validation budgets, how do agents compare with NSGA-II? In particular, can the authors report repeated equal-size subsampling for both fronts and uncertainty for their hypervolume difference?
Thank you for identifying the distinction between overall rollout performance and performance at a common search budget. We have added analyses addressing search budgets, exclusions, and validation.
Q1. Our primary analysis estimates the overall effect of the tool interface under a common four-hour limit. We omit realized resource usage from the primary hurdle model because it is a post-treatment variable. To address the reviewer's efficiency question, Fig. 7 shows hypervolume against evaluation count, token usage, and elapsed wall time for PLI design and electrolyte screening. Table 8 uses the first N evaluations of each trajectory that reached the indicated count and reports hypervolume conditional on agent success. Table 6 gives the counts of eligible runs with both zero and positive hypervolume, alongside the exclusions. We interpret the fixed-count comparisons descriptively because the mix of configurations varies with the evaluation count.
Adding realized resource usage to the hurdle model improved predictive fit primarily for rollout success and added little to our ability to predict final hypervolume among successful runs. Table 8 also shows how far individual PLI searches extended. The largest direct-calling run evaluated 136 designs, compared with 2,191 for PTC. This difference is consistent with greater search volume contributing to the overall PTC advantage.
Q2. Table 6 reports exclusions by task, LLM, harness, tool interface, and data access. We excluded 153 of 774 PLI runs and 10 of 234 electrolyte runs. As a descriptive sensitivity check, counting every excluded run as unsuccessful gives PLI success rates of 39.7% for PTC (196/494) and 23.6% for direct calling (66/280). For electrolyte screening, the rates are 44.0% (59/134) and 23.0% (23/100), respectively. The direction of the unadjusted comparison persists in both tasks. We have not refit the adjusted hurdle model under this alternative rule.
Q3. We allocated 100 PAGOSA simulations to agent designs and 100 to NSGA-II designs, doubling the original validation cohort. Simulations completed for 99 agent designs and all 100 NSGA-II designs. The measured hypervolumes were 8.07 and 8.55, respectively, compared with 5.53 for the historical front (Fig. 3). The revised Methods describe how we selected designs from each group.
To compare search performance at a common evaluation count, we used the expanded validation data to calibrate JANUS predictions against PAGOSA. At 500 evaluations, median predicted hypervolume was 8.22 for the five successful PTC runs with historical data that reached that count (95% HDI 7.35–9.17) and 8.43 for ten NSGA-II runs (7.53–9.52). Figure 7 and Table 8 show how the comparison varies with evaluation count.