- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework for balancing Search, which consumes computational resources, and Experiment, which consumes experimental resources, during scientific discovery. It evaluates the framework on NewtonBench and other benchmarks.
Critical strengths
- The figures are well designed.
- The research question is important to building automated laboratories.
Critical weaknesses
Limited novelty: The method relies on common components, including a meta-controller. Despite the name “CoREAgent,” the paper does not demonstrate clear algorithmic advances in agent design. Its limited action space and lack of explicit task planning also raise questions about whether it qualifies as an agent.
Questions for authors
- Could the authors compare against LLM-assisted Bayesian optimization methods such as LLAMBO, LGBO, LLINBO, and LABO?
- Could the evaluation benchmarks have appeared in the LLM’s training data? Potential contamination makes it difficult to assess the LLM’s contribution to scientific discovery.
- Could the authors provide a systematic component-wise ablation of the full framework to justify its design?
- Could the authors conduct more evaluations in high-dimensional settings, which are common in scientific applications?
- Could the authors better explain the motivation behind the methodological choices?
- Could the authors provide theoretical analysis establishing bounds or guarantees for the method?
- Could the authors evaluate different LLMs, prompt sensitivity, and the effect of reasoning capabilities? GPT-4o-mini is a relatively old and less capable model. With more advanced but expensive models, how would API costs affect resource allocation and overall performance?
9 comments
Response to Reviewer 5uH7 (2/3)
Potential benchmark contamination makes it difficult to assess the LLM’s contribution.
We agree that contamination is important and cannot rule out pretraining exposure. However, our benchmarks incorporate safeguards that reduce opportunities for direct recall.
-
Benchmark design. NewtonBench employs counterfactual physical laws; ForceBench requires inferring unknown interaction laws and experimental control roles; ActiveSciBench-Chem conceals the relevant variables, functional forms, and parameters; and ODEBench removes semantic descriptions. Every task begins with a single seed experiment and requires further evidence acquisition. These safeguards mitigate direct recall without guaranteeing contamination-free evaluation.
-
Empirical evidence. All methods use the same backbone and task descriptions. If recall drove performance, the LLM-only baseline would already solve these tasks; instead, it recovers only 63.0% on NewtonBench and 36.7% on ODEBench. Under identical budgets, CoREAgent raises this to 88.0% and 84.4% (+25.0 and +47.7 points; Sec. 5.2, Table 1), indicating that the gains come from the framework built around the LLM, which generates, tests, and refines hypotheses using experimental evidence.
We have included this clarification in Appendix A.1 (Datasets), acknowledging potential pretraining exposure and explaining the scope of our evaluation.
Evaluations in high-dimensional settings
We agree that high-dimensional evaluation is important and an active area of research. However, realistic, high-dimensional benchmarks supporting interactive scientific discovery and mechanism recovery remain limited. We thus evaluate on four established benchmarks spanning diverse scientific domains, including seven-input enzyme kinetics (ActiveSciBench-Chem) and multidimensional dynamical systems (ODEBench) and physics laws. These evaluations establish performance across heterogeneous experimental interfaces. We acknowledge this limitation in Sec. 7.
Importantly, CoREAgent is not inherently restricted to low-dimensional problems, its metareasoner operates on discovery-state features rather than a fixed-dimensional experimental input, and its hypothesis-search and acquisition modules can accommodate different experimental interfaces. Nevertheless, scalability of these underlying modules in substantially higher dimensions remains to be established. We acknowledge this and identify broader high-dimensional evaluation as future work.
Motivation behind the methodological choices
CoREAgent addresses a central challenge in scientific discovery: further hypothesis refinement cannot resolve ambiguities caused by insufficient evidence, while premature experimentation wastes scarce experimental resources. This motivates three complementary components.
-
Diverse hypothesis search (Sec. 3.1): Exploration–exploitation populations, LLM-guided proposals, symbolic fitting, and cross-population exchange maintain structurally distinct candidate mechanisms, preventing premature convergence.
-
Hypothesis-guided acquisition (Sec. 3.2): Belief-weighted information gain targets experiments that discriminate among competing mechanisms rather than merely reducing predictive uncertainty.
-
Online metareasoning (Sec. 3.3): Bayesian linear value models and Thompson sampling support online allocation decisions. Prequential credit assignment evaluates earlier decisions using subsequently acquired evidence, providing a measurable signal for learning when additional refinement or experimentation is more useful.
Together, these components enable the framework to learn not only which hypotheses to investigate and where to experiment, but when to allocate resources between the two. The ablations (Fig. 4) support their complementary roles, while the cost-sensitivity study (Fig. 3) demonstrates adaptive allocation behavior.
Evaluate different LLMs, prompt sensitivity, and the effect of reasoning capabilities. How would API costs affect resource allocation and overall performance?
We thank the reviewer for these suggestions.
Different LLMs. We evaluated three backbones (GPT-4o-mini, Qwen3-32B, and Claude Haiku 4.5) in App. C.3, Table 7. Recovery varies with the backbone, but experiment usage stays similar, indicating that the metareasoner's allocation behavior is stable across models.
API costs and allocation. We vary the search cost up to 30× and the experiment cost up to 40× (Sec. 5.3, Fig. 3). CoREAgent shifts from whichever action becomes more expensive while maintaining recovery, compared with 78% for LLM-AutoSciLab (see our response to WgsQ). Learned allocation also directly reduces LLM usage: with the same search engine and stopping rule, it reaches comparable recovery with 21.6% fewer LLM calls (Table 8; see our response to yjqQ).
Prompt sensitivity. We could not complete the experiments on these within the rebuttal period and will include them in the camera-ready version.
Response to Reviewer 5uH7 (3/3)
Theoretical analysis establishing bounds or guarantees for the method
We agree that theoretical guarantees would strengthen the work. Our contribution is a sequential decision formulation (Eq. 4) with an online learning procedure, not a regret or optimality guarantee; only budget feasibility is guaranteed, by construction. Guarantees are hard to derive because search changes the set of hypotheses, experiments change the evidence, and feedback arrives late. For the same reason, the prequential credit (Eq. 10) is a practical, short-horizon proxy for an action's value, not an unbiased estimate. We therefore support the method empirically; allocation responds to changing costs (Fig. 3), and learned allocation improves on the ablations, matched controls, and simpler switching rules (Fig. 4, Tables 8–10). Regret bounds for this setting remain an open direction.
Response to Reviewer 5uH7 (1/3)
Thank you for dedicating your time and expertise to review our submission. Please find our responses below.
We thank the reviewer for this concern. Our novelty lies not in introducing a meta-controller, but in what the controller learns to decide and how it learns from experimental evidence.
vs. Active Discovery: These methods refine hypotheses and select informative experiments, largely neglecting acquisition costs. However, these methods do not learn when further hypothesis refinement is more valuable than acquiring new evidence; their search–experiment allocation follows fixed schedules or rules.
vs. Budget-Aware Methods: These methods allocate computational resources or select models for a given task. CoREAgent addresses a distinct challenge: experiments generate new evidence that can invalidate existing hypotheses and change the value of subsequent search. Thus, the allocation decision depends not only on computation, but also on how acquiring new evidence reshapes the discovery process. CoREAgent formalizes this interdependent decision as a sequential allocation problem (Eq. 4) and introduces online prequential credit assignment (Eq. 10), learning the value of both actions from predictive improvements on subsequently acquired observations.
CoREAgent builds on discovery agents such as MDA and DP-Agent, where planning takes the form of hypothesis generation and experiment design, and adds a learned decision on top about when to search and when to experiment.
Empirical Evidence. With the shared hypothesis-search engine (Sec. 5.2, Table 2), CoREAgent achieves comparable recovery on NewtonBench while using only 40.3% of the search and 69.5% of the experimental budget. On ODEBench, it improves recovery from 80.7% to 84.4% with fewer resources. Under matched search, acquisition, and stopping (App. C.4, Table 8), it additionally achieves comparable recovery with 21.6% fewer LLM calls than a fixed schedule.
Thus, our central contribution is learning how to orchestrate hypothesis refinement and evidence acquisition as scientific understanding evolves, rather than introducing novelty in every underlying component or claiming a new general-purpose planning architecture.
We thank the reviewer for highlighting these relevant works. While they share our motivation, their objective is optimizing a scalar function, whereas CoREAgent aims to identify the governing scientific mechanism. This distinction changes both the role of the LLM and the decisions being optimized.
LLAMBO uses LLMs for warm-start initialization, surrogate modeling, and candidate sampling.
LGBO incorporates LLM-generated preferences into the Bayesian surrogate to guide optimization.
LLINBO combines LLM-proposed candidates with statistical surrogate guidance, determining when to trust LLM suggestions versus conventional BO.
LABO adaptively selects between inexpensive LLM-based evaluations and costly experiments using an evolving uncertainty-based gating criterion.
The key distinction from LABO is the decision being learned. LABO selects between evaluation fidelities to improve objective optimization. CoREAgent instead decides between refining competing scientific hypotheses using existing evidence and acquiring new observations to distinguish them. Hypothesis refinement does not itself produce new experimental observations; its value is assessed through subsequent evidence using online prequential credit assignment (Eq. 10).
Our evaluation also includes BED, LLM+BED, and MDA, which directly target Bayesian experiment selection and law recovery (Sec. 4; Sec. 5.1, Fig. 2). We have now added LLM-assisted Bayesian optimization and its distinction from mechanism discovery in Sec. 6.
Fig. 4 (Sec. 5.4) ablates each component, now over three seeds. CoREAgent outperforms every ablation (using the fewest experiments (67.1%) and search calls (49.7%)): removing the metareasoner, guided acquisition, or feedback drops accuracy from 88.9% to 82.2%, and removing diversity drops it to 80.0%.
We added controls to understand the benefits of the learned allocation (see response to reviewer yjqQ and WgsQ):
These results show that every component is therefore necessary, and together they achieve the highest accuracy at the lowest cost.