acceptodds
Under review as a conference paper at ICLR 2027

SelfAI: A Self-Directed Multi-Agent Framework for Long-Horizon Scientific Discovery

Abstract

LLM agents increasingly explore scientific problems on their own, yet they are judged almost solely by the final result they report. That result cannot distinguish an efficient investigation from one that spent most of its budget on redundant trials. We introduce SelfAI, a self-directed multi-agent system that treats experimentation as trajectory-driven decision making. Given a research objective and experimental scope, SelfAI formalizes the task, reasons over accumulated trajectories to plan subsequent trials, and decides when further trials are no longer worthwhile. To evaluate these decisions directly, we build a controlled benchmark in which all methods face the same executed experiments and observe each outcome only after selecting it. Across tasks spanning scientific computing, machine learning, computer vision, medical image analysis, and drug discovery, SelfAI finds the best configurations nearly as reliably as full-budget classical optimization methods and LLM baselines while using substantially fewer trials, whereas early-stopping baselines with similar trial counts miss them more often. Ablations show that trajectory analysis, planning, and stopping make complementary contributions. The evaluation further reveals that methods with similar final quality can differ substantially in how many trials they spend and when they stop. Code and VSCode extensions will be released.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agentic AI, sciencific discovery, decision making, multi-agent systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MVoVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces SelfAI, a multi-agent framework for scientific experimentation that aims to improve the efficiency of experimental exploration. The framework consists of a User Agent for task specification, a Cognitive Agent for analyzing previous trials, planning subsequent experiments, and deciding when to stop, and an Experiment Manager for execution and feedback collection. The authors also introduce a controlled benchmark and two metrics, Score and AUPD, to evaluate experimental trajectories beyond final performance. Experiments across several domains show that SelfAI can identify high-performing configurations with fewer trials than the compared baselines.

Critical strengths
  1. The paper addresses a practically relevant problem in autonomous scientific experimentation. Evaluating how efficiently an agent explores experimental configurations, rather than focusing solely on final performance, is meaningful, particularly when experiments are expensive.

  2. The controlled benchmark provides a useful setting for comparing exploration and stopping strategies under identical experimental outcomes. The authors consider a broad range of classical and LLM-based baselines, and the results demonstrate that SelfAI can reduce the number of trials while maintaining a competitive hit rate. The ablation studies also help establish the contributions of individual components.

Critical weaknesses
  1. The paper does not sufficiently establish the contribution of SelfAI relative to existing research-agent and optimization approaches. The motivation emphasizes that previous systems primarily focus on final outcomes, but prior work such as AIDE and AIRA has already explored using accumulated experimental feedback to guide subsequent search decisions. More importantly, the core functionality of SelfAI's Cognitive Agent, as described in Section 3.1 and Appendix A.3, is largely implemented through structured prompts for analyzing previous trials, identifying promising configurations, and deciding when to stop. While these are reasonable design choices, they resemble established practices in iterative optimization and research-agent workflows. The ablation studies show that these components are useful, but it remains unclear what SelfAI adds beyond combining these existing operations into a multi-stage prompting framework.

  2. The evaluation does not fully isolate the benefits of the proposed framework. For example, SelfAI uses detailed prompts for trajectory analysis, planning, and stopping, whereas the LLM-ES baseline relies on much simpler instructions. It is unclear whether the improvements arise from the multi-stage framework or from providing more explicit reasoning guidance. In addition, the proposed Score metric combines endpoint quality, discovery time, and stopping time using a particular weighting scheme, without sufficient justification for this choice.

Questions for authors
  1. Could the authors clarify what specific limitations of existing history-guided research agents, such as AIDE and AIRA, SelfAI addresses? In particular, what additional capabilities does the proposed framework provide beyond combining trajectory analysis, planning, and stopping through structured prompts?

  2. How well do the findings from the controlled benchmark generalize to open-ended scientific discovery? Given the mixed results on MLE-Bench, could the authors provide further evidence or clarify the intended scope of SelfAI's applicability?

  3. Would a single-agent baseline with equivalent reasoning instructions and inference budgets achieve similar results? Also, does SelfAI maintain its advantages under standard anytime evaluation metrics or different stopping criteria?

5 comments

We thank the reviewer for pressing on what SelfAI adds and on whether the gains come from prompt detail. We further checked all runs behind the tables, and as a result, corrected the setup of several baseline runs on four problems (GraphSAGE, LSTM sentiment analysis, Boston and LCBench) and of two ablation runs. We also corrected the trial counts for several runs in which the same configuration was counted more than once. We replace the corresponding values in Tables 2–5 and the supplementary tables of the revised paper. These corrections do not change the paper’s main conclusions.

Q1. Limitations of AIDE and AIRA that SelfAI addresses and capabilities beyond structured prompts. We thank the reviewer for this question.

Limitations of AIDE and AIRA that SelfAI addresses. Both use feedback inside a fixed search procedure. A hand-designed rule picks the next node (greedy selection in AIDE, and greedy, MCTS or evolutionary selection in AIRA), the LLM acts as an operator that drafts, debugs or improves that node, the search runs until a fixed step or time budget, and the agent is judged by the quality of the final solution. SelfAI addresses these three limitations.

  1. A predefined search policy. AIDE and AIRA choose the next step by a predefined search policy (greedy improvement, or MCTS with UCT selection), while SelfAI plans from its analysis of the trajectory, as the Cognitive Agent analyzes all completed trials and plans which hypotheses to test next. Under the same SelfAI stopping judgement, replacing this trajectory-based selection with LLM expansion plus UCT selection (AIRA-style) lowers Score from 0.6019 to 0.3689 and hit rate from 0.9231 to 0.6154 (Table 5). An AIDE-style greedy policy, which improves the current best configuration and occasionally drafts a random one, needs 18.15 trials and reaches Score 0.4890 and hit rate 0.8462.
  2. A fixed budget. AIDE and AIRA search until a fixed step or time budget is spent. SelfAI instead decides when to stop from the same trajectory analysis. Without this decision, SelfAI reaches the same endpoint quality (Best Norm 1.000 vs. 0.998) only with 35.92 instead of 12.92 trials. With a patience rule on the same search policy (Rule-SelfAI), it needs 27.15 trials and hits fewer problems (Table 4).
  3. Evaluation by the final solution only. In AIDE- and AIRA-style evaluation, the endpoint mixes code quality, execution and validation with search decisions. Our controlled benchmark fixes the outcomes, so only selection and stopping differ, and the trajectory metrics show that methods with near-identical endpoint quality differ by about 3x in trials spent.

Capabilities beyond combining these functions through structured prompts. In addition to combining these functions through structured prompts, the framework offers other capabilities. The User Agent translates natural-language goals, ideas and constraints into task specifications for users to review and modify before execution. The Experiment Manager runs experiments, handling environment preparation, resource coordination, progress tracking, checkpoint-based recovery and failure reporting, and adds each result to the experimental history (Sect. 3). Furthermore, the framework operates independently of the native agent and runs inside the workflows of AIDE and ML-Master (Supplementary B.7).

W2-1. Prompt detail or framework? We thank the reviewer for this comment. As the reviewer notes, the LLM-ES prompt is much shorter than SelfAI's stopping prompt. We have corrected its description in the Supplementary and Table 4. LLM-ES has the same inputs and stop-or-continue action as SelfAI, but not SelfAI's analysis and planning context or its stopping criteria. LLM-ES+ receives SelfAI's full three-criterion stopping prompt and reaches a hit rate of 0.6923, vs. 0.9231 for SelfAI at a similar number of trials (13.46 vs. 12.92, Table H). LLM-MCTS + SelfAI stop uses the same stopping prompt and reaches 0.6154 (Table 5), and the LLM baseline receives SelfAI's planning rules verbatim yet needs 33.38 trials, vs. 12.92 for SelfAI.

Table H. Baselines given parts of SelfAI's prompts (mean over 13 problems, values from Tables 4, 5 and F). TA denotes trajectory analysis, PR SelfAI's planning rules and SP SelfAI's stopping prompt.

MethodTAPRSPScore↑AUPD↓t_best↓t_stop↓Best Norm↑Hit Rate↑#Trials↓
LLM✗✓✗0.30790.99210.40020.98401.00001.000033.38
LLM-ES✗✗✗0.43700.49840.55290.56220.99030.692317.62
LLM-ES+✗✗✓0.49420.46930.49810.51010.99480.692313.46
LLM-MCTS + SelfAI stop✗✗✓0.36890.60190.62660.60600.95280.615418.08
SelfAI✓✓✓0.60190.33760.36420.43130.99810.923112.92

W2-2. Justification of the Score weighting. We thank the reviewer for this comment. Score multiplies endpoint quality (Gain) by one minus the mean of two costs, the delay before the best result is found (t_best) and the trials spent before stopping (t_stop). The product ensures that a fast but poor run cannot score highly. Equal weights reflect that, without a task-specific cost model, we have no basis to value one cost above the other. If the weight is varied, Score_w = Gain · (1 − w·t_best − (1 − w)·t_stop), SelfAI ranks first for every w in [0, 1] (Table G). Even at w = 0, where the metric rewards only early stopping, SelfAI stays ahead of BS-ES, which stops earliest in Table 3 (t_stop = 0.33) but has a hit rate of 0.3077.

Table G. Score_w for all Table 3 methods (mean over 13 problems). w = 0.5 gives the Score of Table 3, w = 0 rewards only early stopping and w = 1 only early discovery.

Methodw = 0w = 0.25w = 0.5w = 0.75w = 1
GS0.00000.11540.23080.34620.4616
BS (TPE)0.02150.10190.18240.26280.3432
SMAC30.14640.25210.35790.46360.5694
Random0.00000.13890.27780.41670.5557
SHA0.00000.12100.24200.36300.4840
MCTS0.00000.13980.27950.41930.5590
LLM0.01600.16190.30790.45380.5998
LLM-MCTS0.00000.11510.23020.34540.4605
OPRO0.00000.13940.27880.41810.5575
SelfAI w/o stopping0.00000.14690.29370.44060.5874
Random-ES0.54040.48080.42120.36160.3021
BS-ES (TPE-ES)0.56270.48830.41390.33950.2652
MCTS-ES0.47550.44070.40600.37120.3365
MCTS-UCT-stop0.10860.21750.32630.43520.5440
LLM-ES0.42690.43200.43700.44210.4471
Rule-SelfAI0.27850.35010.42170.49330.5648
LLM-MCTS-ES0.44640.39490.34340.29190.2404
SelfAI0.56790.58490.60190.61880.6358

Q2. Generalization and intended scope. Generalizing the findings from our controlled benchmark to open-ended scientific discovery depends on three conditions: whether the methods follow a common experimental protocol, whether the outcomes of candidate experiments reliably reflect the final objective, and whether the same code-generation procedure is used when comparing search and stopping strategies. Our current evidence shows that SelfAI’s exploration and stopping mechanisms can be applied to open-ended discovery tasks with AIDE and ML-Master. When these three conditions are met in open-ended agents, we expect SelfAI’s gains in experiment selection and stopping efficiency to carry over to open-ended discovery tasks.

Intended scope. SelfAI is intended for experimental processes with four properties. Each experiment is costly, outcomes can be compared using an evaluation metric, the number of experiments is limited by a budget, and the task can be expressed as a specification containing an objective, admissible experimental choices, evaluation criteria, and a budget. Under these properties, decisions based on the accumulated trajectory about which experiment to run next and whether to continue determine how the budget is allocated and what is ultimately discovered. In open-ended scientific discovery, where a native agent generates and executes the candidates, we expect SelfAI’s gains to carry over when the resulting experimental trajectory is sufficiently rich and reliable—that is, when the native agent produces enough runnable candidates and their validation outcomes reliably reflect the final objective. Such a trajectory provides a reliable evidence base for trajectory analysis and stopping decisions.

Q3. Single-agent baseline, anytime metrics, and other stopping criteria. We thank the reviewer for these questions. SelfAI keeps its advantage in all three comparisons.

Single-agent baseline. A single agent with the same instructions, history and backbone does not achieve similar results (setup and Table D in our response to Reviewer 1, Q3).

Standard anytime metrics. At a fixed trial budget equal to SelfAI's trial count on each problem, every full-budget baseline has a lower hit rate and Best Norm than SelfAI. Matching the patience of the ES baselines to SelfAI's mean trial count leads to the same conclusion (Table C in our response to Reviewer 2, Q3).

Table E. Full-budget baselines cut at SelfAI's trial count on each problem (12.92 trials on average).

MethodScoret_bestHit rateBest Norm
BS (TPE)0.34630.83680.30770.9902
Random0.44390.62310.50770.9778
SMAC30.47860.56810.61540.9894
SHA0.41370.66630.38460.9124
MCTS0.47600.57480.61540.9854
LLM-MCTS0.38420.74030.38460.9448
OPRO0.43690.64720.53850.9908
LLM0.48460.58440.53850.9837
SelfAI0.60190.36420.92310.9981

Different stopping criteria. The baselines in Table 3 stop by patience (Random-ES, BS-ES, MCTS-ES, LLM-MCTS-ES), by a UCT-based criterion (MCTS-UCT-stop) or by an LLM stopping judgment (LLM-ES), and SelfAI has the highest Score among them. On the same SelfAI search policy, its stopping judgment (Score 0.6019, 12.92 trials) also outperforms patience stopping (0.4217 and 27.15 trials for Rule-SelfAI) and no stopping (0.2937, 35.92 trials).

Reviewer_FGRqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents SelfAI, a multi-agent framework that formalizes a user objective, analyzes completed trials, selects subsequent experiments, and decides when to stop. Its main evaluation uses fixed, pre-executed candidate tables. At each round, the agent selects an unevaluated model configuration or hyperparameter combination and observes its scalar objective value. The paper introduces Score and AUPD to evaluate endpoint quality, time to the reference-best configuration, stopping time, and the distribution of observed outcomes. SelfAI reportedly finds near-optimal configurations using fewer trials than full-budget search methods. Ablations attribute these gains to trajectory analysis, strategic planning, and stopping. A separate MLE-Bench evaluation applies SelfAI to action selection and stopping within existing ML agents, with mixed results.

Critical strengths
  1. The controlled benchmark offers a useful way to isolate configuration selection and stopping from code-generation and execution quality.
  2. The component ablations provide evidence that trajectory analysis and planning contribute beyond early stopping alone.
Critical weaknesses
  1. The framing exceeds what the evaluation establishes. The controlled benchmark is a reasonable way to isolate configuration selection and stopping. However, its tasks involve selecting from fixed, pre-executed configuration tables with immediate scalar feedback, closely resembling sequential hyperparameter optimization. Defining “long-horizon” as using accumulated history does not establish what additional planning capability is tested. The supplementary MLE-Bench results provide mixed evidence for broader applicability. The paper should either sharpen its contribution within configuration search and stopping or provide stronger evidence supporting its broader scientific-discovery framing.
  2. It is unclear whether experimental costs are counted consistently across methods. Section 3.2 states all methods use a common evaluation horizon. However, the timing values in Table 10 appear normalized by 20 configurations for grid search and 17 for other methods. Table 11 shows a similar pattern of 64 versus 61. This suggests the three initial trials may be treated differently across methods, but the paper does not explain this convention. Since the main claim concerns reducing the number of experiments, the authors need to clarify this accounting and confirm that efficiency comparisons use a consistent basis.
Questions for authors
  1. What specific capability does long-horizon refer to beyond history-dependent configuration selection, and which experiments demonstrate that capability?
  2. Are the three initial trials counted consistently across methods? Please explain the different denominators suggested by Tables 10–11.
  3. How were the baselines' stopping rules configured? Does SelfAI retain its advantage when the baselines are allowed a comparable number of trials?
4 comments

We thank the reviewer for identifying the denominator inconsistency, which we have corrected, and for the clear suggestion on framing. We further checked all runs behind the tables, and as a result, corrected the setup of several baseline runs on four problems (GraphSAGE, LSTM sentiment analysis, Boston and LCBench) and of two ablation runs. We also corrected the trial counts for several runs in which the same configuration was counted more than once. We replace the corresponding values in Tables 2–5 and the supplementary tables of the revised paper. These corrections do not change the paper’s main conclusions.

W1. Contribution and evidence. We thank the reviewer for this comment. Both scientific discovery and HPO involve searching for high-performing configurations within a limited budget. However, SelfAI asks how an agent uses the accumulated experimental trajectory to identify promising directions, decide which experiments to run next, and judge whether further trials are worthwhile given the remaining budget during scentific discovery. This encompasses the entire experimental process managed by the agent: from interpreting the task and analyzing experimental trajectories to formulating hypotheses, planning experiments, and determining when to conclude the experimental process.

Q1. Capability beyond history-dependent selection, and the experiments that test it. We thank the reviewer for this question. Beyond history-dependent configuration selection, long-horizon refers to budget-aware planning over the whole experimental trajectory, which means deciding not only which experiment to run next, but also whether further experiments are still worth their cost, with both decisions made from the accumulated trajectory rather than from the latest result. History-dependent selection answers only the first question. We have stated this in the definition added to Sect. 1. The experiments that demonstrate this capability are (Table F):

Stopping judgement.

  • the stopping comparison on the same search policy: SelfAI's trajectory-based stopping judgement vs. patience stopping (27.15 trials and hit rate 0.8462 for SelfAI w/ patience stopping, vs. 12.92 trials and 0.9231 for SelfAI, Table 4).
  • stopping criteria without trajectory analysis (LLM-ES+): given SelfAI's three stopping criteria but neither trajectory analysis nor strategic planning, the model uses 13.46 trials but finds the best configuration on only 69% of the problems (hit rate 0.6923), vs. 92% (hit rate 0.9231) for SelfAI.
  • a new history-truncation experiment, in which SelfAI sees only its last k trials. With k = 1 it rarely stops (2 of 13 problems, 23.00 trials). With k = 3 or 5 it stops, but hits fewer problems (0.8462 and 0.6154). Only the full history reaches a hit rate of 0.9231 within about 13 trials, so the decisions depend on the whole trajectory.

Strategic planning.

  • planning removed (Table 4): with the same analysis and stopping judgement, SelfAI without strategic planning stops at a similar point (13.38 trials) but finds the best configuration on only 54% of the problems (hit rate 0.5385) instead of 92% (hit rate 0.9231), and later (t_best 0.62 vs. 0.36).
  • planning replaced by a fixed search rule: under the same stopping judgement, LLM expansion with UCT selection finds the best configuration on 62% of the problems (hit rate 0.6154, Score 0.3689, Table 5), and an AIDE-style greedy policy needs 18.15 trials (Score 0.4890).

Table F. New experiments, in the format of Table 4 (mean over 13 problems). TA, SP, SJ and PS as in Table 4. SelfAI w/ history k sees only its last k completed trials. LLM-ES+ is LLM-ES given SelfAI's stopping criteria.

MethodTASPSJPSScore↑AUPD↓t_best↓t_stop↓Best Norm↑Hit Rate↑#Trials↓
SelfAI✓✓✓✗0.60190.33760.36420.43130.99810.923112.92
LLM-ES+✗✗✓✗0.49420.46930.49810.51010.99480.692313.46
SelfAI w/ history k = 1✓✓✓✗0.32500.85720.49180.85680.99990.923123.00
SelfAI w/ history k = 3✓✓✓✗0.56060.42270.43430.44420.99990.846213.08
SelfAI w/ history k = 5✓✓✓✗0.58310.18930.54070.27010.99020.61549.38
AIDE-greedy + SelfAI stop✗✗✓✗0.48900.57800.43890.57760.99350.846218.15

The planning capability we test is the agent’s ability to turn patterns in earlier results into hypotheses about which untested configurations are worth evaluating and to use those hypotheses to guide subsequent trials. In Table 4, removing strategic planning while retaining trajectory analysis and stopping judgement reduces the hit rate from 12/13 to 7/13, with similar trial counts (12.92 vs. 13.38). In a separate comparison, replacing SelfAI’s stopping judgement with a patience-based rule while retaining its search policy increases the mean trial count from 12.92 to 27.15 and reduces the hit rate from 12/13 to 11/13. These comparisons test how the agent allocates trials and decides when to stop as evidence accumulates.

MLE-Bench. We use the MLE-Bench to illustrate the challenges facing current open scientific discovery workflows. This prompted us to build a controlled benchmark that compares how methods allocate trials and decide when to stop under the same experimental design. We clarify our contribution as improving configuration search and adaptive stopping within a given set of candidate experiments:

The Cognitive Agent uses hypotheses drawn from earlier trials to prioritize which candidate experiments to test next and assesses whether any remaining candidates are promising enough to justify further trials within the budget.

Q2. Are the three initial trials counted consistently? We thank the reviewer for pointing out this issue. We have checked all tables thoroughly, and every method now uses H_i = |R_i| − 3. #Trials, in contrast, includes the initial trials, consistently for all methods, so the efficiency comparisons between SelfAI and the baselines were already on a consistent basis. We have updated Tables 3 and 9–20 accordingly and stated the accounting rule explicitly in Sect. 3.2.

Q3. Baseline stopping rules and comparison at matched trial counts. We thank the reviewer for this question.

Configuration. All baselines start from SelfAI's three initial trials and use the same settings on every problem. Random-ES, BS-ES, MCTS-ES and LLM-MCTS-ES stop after p = 5 consecutive trials that do not improve the best value so far. Rule-SelfAI uses the same patience and also requires the best value to be 10% better than the mean of the first three trials. MCTS-UCT-stop runs the same heuristic MCTS as MCTS-ES. It stops when the largest UCT exploration bonus at the root, λ0(1 − t/B)·sqrt(ln(N_root + 1)/N_child) with λ0 = 0.1, falls below 0.02. Because this bonus shrinks with the remaining budget, MCTS-UCT-stop stops late (32.69 trials on average).

Comparable trial counts. SelfAI keeps its advantage when the baselines receive SelfAI's trial budget. We matched the trial counts in two ways. First, with p = 5 the ES baselines stop earlier than SelfAI (8.92–10.15 trials vs. 12.92). To remove any dependence on this choice, we swept p from 1 to 15 for every ES baseline (Table C). Where the mean trial count is closest to SelfAI's 12.92, the hit rates are 0.15–0.62, against 0.92 for SelfAI. To reach SelfAI's hit rate, Random-ES and MCTS-ES need 20.5–21.8 trials, about 1.6 times SelfAI's 12.92, and BS-ES and LLM-MCTS-ES never reach it, staying at 0.85 even with about 21 trials. Second, when every full-budget baseline is given SelfAI's number of trials on each problem, the best of them, SMAC3 and MCTS, find the best configuration on 62% of the problems (hit rate 0.6154), against 92% (hit rate 0.9231) for SelfAI. Every baseline also has a lower Score (at most 0.4846, against 0.6019), finds the best configuration later (t_best 0.5681–0.8368 vs. 0.3642) and reaches a lower Best Norm (at most 0.9908, against 0.9981). Conversely, baselines that use more trials than SelfAI still find the best configuration less often. LLM-ES uses 17.62 trials and finds it on 69% of the problems (hit rate 0.6923), and Rule-SelfAI uses 27.15 trials and finds it on 85% (hit rate 0.8462).

Table C. Mean trial count and hit rate for odd p values from 1 to 15. For each p, we apply the patience rule to the recorded full-budget runs and use the trials up to the resulting stopping point.

pRandom-ESBS-ESMCTS-ESLLM-MCTS-ES
14.31 / 0.094.23 / 0.004.23 / 0.004.23 / 0.00
36.20 / 0.206.08 / 0.006.46 / 0.315.69 / 0.08
59.71 / 0.428.54 / 0.0810.15 / 0.4610.46 / 0.46
712.65 / 0.5210.38 / 0.0812.00 / 0.4613.54 / 0.62
915.83 / 0.6612.92 / 0.1514.54 / 0.6216.15 / 0.69
1117.98 / 0.7518.23 / 0.4615.92 / 0.6217.38 / 0.69
1319.89 / 0.8321.46 / 0.8519.31 / 0.8519.92 / 0.85
1521.83 / 0.9222.08 / 0.8520.77 / 0.9220.69 / 0.85

Thank you for the clarifications and additional experiments. The corrected trial accounting addresses my concern about evaluation consistency, and the comparisons at matched trial counts strengthen the evidence for the method’s search efficiency. I also appreciate the additional ablations examining the use of experimental history and the stopping mechanism.

My remaining concern is the scope of the contribution. The additional ablations demonstrate the value of accumulated experimental history, but remain confined to fixed, pre-executed candidate tables. These results support history-dependent selection and adaptive stopping within a given candidate set. However, with the MLE-Bench results remaining mixed, I am still not convinced that the evidence sufficiently supports the broader claims of long-horizon scientific discovery. The rebuttal has improved my assessment, but this concern remains that I am keeping my current rating.

Reviewer_ArMbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SelfAI, an LLM-based multi-agent framework for scientific experimentation that formalizes research objectives, analyzes previous experimental results, plans subsequent trials, and decides when to stop. The authors develop a benchmark with fixed candidate configurations and precomputed outcomes, together with evaluation metrics that capture final performance, discovery and stopping times, and the diversity of observed outcomes.

Critical strengths
  1. The paper addresses an important problem: how research agents should use accumulated experimental evidence to select subsequent trials and decide when to stop. Its controlled benchmark fixes candidate configurations and experimental outcomes across methods, providing a useful design for studying these decisions while reducing confounding from code generation and execution reliability. Evaluating final quality together with discovery time, stopping time, and trial counts also makes the trade-off between performance and experimental effort explicit.

  2. The study separately removes trajectory analysis, strategic planning, and stopping, and compares different search policies under the same stopping mechanism. These comparisons provide useful evidence about how the components affect search behavior.

Critical weaknesses
  1. There are inconsistencies between the evaluation settings and reported results. The paper states that all methods share the same finite candidate space and unique-trial budget, and that AUPD is invariant to trial ordering. However, Table 3 reports that MCTS and OPRO exhaust the full budget but achieve only a 92.31% hit rate, whereas Grid Search reaches 100%. Similarly, SelfAI without stopping achieves an AUPD of 0.810 compared to 1.000 for Grid Search, despite both apparently evaluating the complete candidate set. The normalization of discovery and stopping times also appears inconsistent across methods in the supplementary tables.

  2. The paper doesn’t have enough evidence to support the broader claims of long-horizon scientific discovery. Most controlled benchmark tasks involve selecting configurations from small, predefined candidate spaces, often containing only 9–30 configurations. This setting isolates experimental selection and stopping decisions, but does not evaluate open-ended hypothesis generation or experiment design.

    On the larger LCBench task, none of the evaluated SelfAI backbones reaches the reference optimum, while the open-ended MLE-Bench experiments show mixed results, with improvements in only three of six private-score comparisons. I think the evidence primarily supports efficient configuration selection in constrained settings rather than the broader applicability of SelfAI to long-horizon scientific discovery.

Questions for authors
  1. How are the stopping decisions made? Appendix A.3 describes qualitative criteria such as whether unexplored configurations are unlikely to improve performance and whether recent improvements are significant. How are these criteria translated into concrete stopping decisions? Are the decision thresholds or confidence scores calibrated, and are the same settings used across tasks?

  2. How much computational overhead does SelfAI introduce? The paper evaluates efficiency primarily through the number of experimental trials. Could the authors report the additional LLM inference calls, token consumption, and wall-clock time required by trajectory analysis, planning, and stopping judgments? It would help to understand whether fewer experimental trials translate into lower overall computational cost.

  3. Is the multi-agent architecture necessary for the reported improvements? While the paper includes ablations of trajectory analysis, planning, and stopping, it is unclear whether separating these functions into multiple agents provides benefits over a single LLM agent with access to the same experimental history, tools, and computational budget. Could the authors clarify the advantages of the multi-agent design or provide a comparison?

5 comments

We thank the reviewer for the precise reading of Table 3 and the supplementary tables. We have fixed the trial-counting issue the reviewer pointed out, and it does not change the main comparison. We further checked all runs behind the tables, and as a result corrected the setup of several baseline runs on four problems (GraphSAGE, LSTM sentiment analysis, Boston and LCBench) and of two ablation runs. We also corrected the trial counts for several runs in which the same configuration was counted more than once. We replace the corresponding values in Tables 2–5 and the supplementary tables of the revised paper. These corrections do not change the paper’s main conclusions.

W1. Full-budget hit rates, AUPD of full-budget methods, and how trials are counted.

Full-budget hit rate of 92.31% (12/13 problems). We thank the reviewer for spotting this. After checking the logs, we agree that a method evaluating the complete candidate set should hit the global optimum. The discrepancy comes from an evaluation error on one problem (GraphSAGE), which affected SHA, MCTS, LLM-MCTS and OPRO. We have corrected this, and all four methods now reach a 100% hit rate. These corrections do not change the paper’s main conclusions.

AUPD of full-budget methods. The AUPD reference included the three initial trials used by all methods, while every other method is scored only on its selections after them. The reference now excludes them as well, following the rule below for how trials are counted. With this change and the corrections above, most methods have AUPD 1.000 on every problem. BS, SMAC3 and LLM stay below 1.000 because they cannot propose new configurations, so they stop or exhaust the budget before covering the complete table. Score, hit rate and #Trials are unchanged, and SelfAI remains first in Score.

#Trial. We thank the reviewer for pointing this out. We have checked all tables thoroughly, and every method now uses H_i = |R_i| − 3. #Trials, in contrast, includes the initial trials, consistently for all methods. We have updated Tables 3 and 9–20 accordingly and stated the accounting rule explicitly in Sect. 3.2.

W2. Does a setting that isolates selection and stopping, without evaluating open-ended hypothesis generation or experiment design, provide evidence for long-horizon discovery?

We thank the reviewer for raising this concern. Our evidence supports SelfAI’s ability to make two decisions in long-horizon scientific discovery under a limited budget: which experiment to run next and when further trials are no longer worthwhile. Together, these decisions determine how the experimental budget is spent. As defined in Section 1, both decisions draw on evidence accumulated across multiple rounds and account for the remaining budget. The controlled benchmark uses predefined candidate spaces, often with 9–30 configurations, to evaluate these decisions under common conditions. Removing strategic planning reduced the number of tasks on which SelfAI found the best configuration from 12 of 13 to 7 of 13 at similar trial counts. Replacing SelfAI’s stopping judgment with a patience rule while keeping the search policy fixed increased the mean trial count from 12.92 to 27.15 (Table 4).

On LCBench, pursuing the exact optimum in this sparsely sampled space could require many additional trials for a small improvement. Based on the results observed during the search, SelfAI achieved 97.0–99.7% of the reference optimum after evaluating only 1.8–8.1% of the 2,000 configurations. Therefore, SelfAI judged that the likely improvement did not justify spending more of the budget. This is a reasonable trade-off when experiments are costly and the goal is to achieve strong performance within a limited budget.

Open-ended hypothesis generation and experimental design concern which new experiments can be proposed and implemented. SelfAI’s decisions concern which available experiment to pursue and when to stop. Their effectiveness in the open workflow depends on whether the coding agent produces consistent and runnable candidates and whether validation results reliably reflect the final objective. Since private-test scores reflect the combined performance of these stages, they cannot isolate the contribution of SelfAI’s two decisions. To address this, we use a controlled benchmark that holds the candidate set and experimental outcomes fixed across methods, so differences in trial use and achieved performance can be attributed to how methods decide which experiment to pursue and when to stop. Our evidence supports SelfAI’s ability to make two decisions in long-horizon scientific discovery. The controlled results show that SelfAI can use evidence accumulated across rounds to find high-performing configurations with fewer trials, directly supporting our claim about how to allocate a limited experimental budget in long-horizon scientific discovery.

Q1. How are the criteria translated into concrete stopping decisions? We thank the reviewer for this question. The stopping call uses the stopping prompt of the paper (Supplementary A.3.2) and receives the completed trials, the list of unexplored candidates and the trajectory analysis from the same round, which names the promising configurations and the observed trends. The prompt asks the agent to judge each of the three criteria with a one- or two-sentence justification and to answer Yes only if all three are met, and the search stops if and only if the parsed answer is Yes.

Supplementary Figs. 5–7 show these decisions on SIREN (initial trials in blue, selections before the optimum in green and selections after it in red). We follow the segmentation run with GPT4-o3-mini (Supplementary Fig. 5) selection by selection. The best initial trial is k2 = 0.1 with lr = 1e-4 (PSNR 15.59). Selections 1–7 test other k2 values across learning rates and all stay below 8.2, so none surpasses the best initial trial and the search continues. Selection 8 (k2 = 1e-4, lr = 1e-3) reaches 15.36 and reveals a second high-PSNR region at small k2 and small learning rate. Selections 9–12 check further k2 values and all stay below 4. The analysis then attributes the high scores to a small learning rate paired with a small k2, and selection 13 refines this region with lr = 5e-4, reaching the optimum (15.66). Selections 14–16 test three remaining combinations (k2 = 0.01 with lr = 1e-4, k2 = 1e-3 with lr = 0.1, and k2 = 5e-4 with lr = 5e-4), and all stay below 8. The analysis now finds that larger k2 values give lower PSNR and that k2 = 1e-4 with lr = 5e-4 remains the best, so no promising configuration is left untested, the remaining candidates are ranked below the best, and the best PSNR is above all initial trials. The stopping judgment answers Yes after 16 selections. LLM-ES, which receives the same trials without the trajectory analysis, reaches the optimum only at selection 17 and continues to selection 22.

The confidence scores are recorded with each answer but are not involved in the final decision. There are no task-specific thresholds, so no calibration is involved. The prompt, temperature (0) and rules are identical across all problems and all backbones.

Q2. Computational overhead. We thank the reviewer for this question. Table A reports the LLM calls, tokens and time of trajectory analysis, planning and stopping, and Table B compares the training time saved by stopping early with this overhead. SelfAI's overhead is a few LLM calls per round, about 1.6 minutes per problem on average, which is small next to model training. In exchange, it skips about two thirds of the trials that full-budget methods run, and each skipped trial costs from seconds to hundreds of GPU hours. Even on the cheapest task, Boston, the training time saved is several times the LLM time.

Table A. SelfAI's LLM overhead per problem, by stage (mean over all 13 problems). Calls and output tokens are counted from the logged responses, input tokens are reconstructed from the prompts, and times are converted from token counts with the per-token latency measured on the same backbone.

StageLLM callsInput tokensOutput tokensTime
Trajectory analysis8.238.2k3.5k52 s
Planning7.6212.9k2.1k31 s
Stopping judgment6.856.5k0.9k13 s
Total22.7027.7k6.5k97 s

Table B. Overall cost per problem: training time saved by SelfAI's early stopping vs. SelfAI's LLM time. Times per trial are from the original executions.

TaskTrials, full budget / SelfAITime per trial (hardware)Training time savedSelfAI LLM time
Boston162 / 62–6 s (1 core, Intel Xeon Platinum 8358P)156 trials, 5–16 min49 s
SIREN (4 problems)25 / 13–19≈5 min (1 × NVIDIA L20)6–12 trials, 30–60 min34–112 s
BraTS (nnU-Net)18 / 12≈1 h (1 × NVIDIA A100 40GB)6 trials, ≈6 h96 s
BTCV (nnU-Net Revisited)19 / 98–233 GPU h (1 × NVIDIA A100 40GB)10 trials, 80 GPU h or more69 s

Q3. Is the multi-agent architecture necessary? We thank the reviewer for this question. We compared SelfAI with a single-agent variant that performs trajectory analysis, stopping judgment and strategic planning in one call per round, with the same prompts, experimental history and backbone. Given the same number of trials as SelfAI on each problem, the single agent finds the best configuration on 54% of the problems (hit rate 0.5385), close to Random search (51%, hit rate 0.5077), vs. 92% (hit rate 0.9231) for SelfAI (Table D). In SelfAI, the stopping judgment is made in its own call from the trajectory analysis, and strategic planning is called only if the search continues. In the single agent, the stopping judgment and strategic planning are answered in one output, and the model keeps selecting new configurations instead of stopping. The multi-agent design therefore suits this setting better than a single agent.

Table D. SelfAI vs. a single agent that merges its analysis, stopping and planning prompts into one call per round.

MethodScoreAUPDt_bestt_stop#TrialsBest NormHit rateHit rate at SelfAI's trial count
Single agent0.35190.78730.50420.787322.620.99400.84620.5385
SelfAI0.60190.33760.36420.431312.920.99810.92310.9231

Comments on the paper

1 comment

Dear Reviewers,

We are nearing the end of the discussion phase. If you haven't already done so, please read the authors' responses, reply to acknowledge their feedback, and indicate whether your initial concerns have been addressed (along with any score adjustments, if warranted).

Thank you for your dedicated efforts and valuable insights!

Best, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.