- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces SelfAI, a multi-agent framework for scientific experimentation that aims to improve the efficiency of experimental exploration. The framework consists of a User Agent for task specification, a Cognitive Agent for analyzing previous trials, planning subsequent experiments, and deciding when to stop, and an Experiment Manager for execution and feedback collection. The authors also introduce a controlled benchmark and two metrics, Score and AUPD, to evaluate experimental trajectories beyond final performance. Experiments across several domains show that SelfAI can identify high-performing configurations with fewer trials than the compared baselines.
Critical strengths
-
The paper addresses a practically relevant problem in autonomous scientific experimentation. Evaluating how efficiently an agent explores experimental configurations, rather than focusing solely on final performance, is meaningful, particularly when experiments are expensive.
-
The controlled benchmark provides a useful setting for comparing exploration and stopping strategies under identical experimental outcomes. The authors consider a broad range of classical and LLM-based baselines, and the results demonstrate that SelfAI can reduce the number of trials while maintaining a competitive hit rate. The ablation studies also help establish the contributions of individual components.
Critical weaknesses
-
The paper does not sufficiently establish the contribution of SelfAI relative to existing research-agent and optimization approaches. The motivation emphasizes that previous systems primarily focus on final outcomes, but prior work such as AIDE and AIRA has already explored using accumulated experimental feedback to guide subsequent search decisions. More importantly, the core functionality of SelfAI's Cognitive Agent, as described in Section 3.1 and Appendix A.3, is largely implemented through structured prompts for analyzing previous trials, identifying promising configurations, and deciding when to stop. While these are reasonable design choices, they resemble established practices in iterative optimization and research-agent workflows. The ablation studies show that these components are useful, but it remains unclear what SelfAI adds beyond combining these existing operations into a multi-stage prompting framework.
-
The evaluation does not fully isolate the benefits of the proposed framework. For example, SelfAI uses detailed prompts for trajectory analysis, planning, and stopping, whereas the LLM-ES baseline relies on much simpler instructions. It is unclear whether the improvements arise from the multi-stage framework or from providing more explicit reasoning guidance. In addition, the proposed Score metric combines endpoint quality, discovery time, and stopping time using a particular weighting scheme, without sufficient justification for this choice.
Questions for authors
-
Could the authors clarify what specific limitations of existing history-guided research agents, such as AIDE and AIRA, SelfAI addresses? In particular, what additional capabilities does the proposed framework provide beyond combining trajectory analysis, planning, and stopping through structured prompts?
-
How well do the findings from the controlled benchmark generalize to open-ended scientific discovery? Given the mixed results on MLE-Bench, could the authors provide further evidence or clarify the intended scope of SelfAI's applicability?
-
Would a single-agent baseline with equivalent reasoning instructions and inference budgets achieve similar results? Also, does SelfAI maintain its advantages under standard anytime evaluation metrics or different stopping criteria?
5 comments
W2-2. Justification of the Score weighting. We thank the reviewer for this comment. Score multiplies endpoint quality (Gain) by one minus the mean of two costs, the delay before the best result is found (t_best) and the trials spent before stopping (t_stop). The product ensures that a fast but poor run cannot score highly. Equal weights reflect that, without a task-specific cost model, we have no basis to value one cost above the other. If the weight is varied, Score_w = Gain · (1 − w·t_best − (1 − w)·t_stop), SelfAI ranks first for every w in [0, 1] (Table G). Even at w = 0, where the metric rewards only early stopping, SelfAI stays ahead of BS-ES, which stops earliest in Table 3 (t_stop = 0.33) but has a hit rate of 0.3077.
Table G. Score_w for all Table 3 methods (mean over 13 problems). w = 0.5 gives the Score of Table 3, w = 0 rewards only early stopping and w = 1 only early discovery.
| Method | w = 0 | w = 0.25 | w = 0.5 | w = 0.75 | w = 1 |
|---|---|---|---|---|---|
| GS | 0.0000 | 0.1154 | 0.2308 | 0.3462 | 0.4616 |
| BS (TPE) | 0.0215 | 0.1019 | 0.1824 | 0.2628 | 0.3432 |
| SMAC3 | 0.1464 | 0.2521 | 0.3579 | 0.4636 | 0.5694 |
| Random | 0.0000 | 0.1389 | 0.2778 | 0.4167 | 0.5557 |
| SHA | 0.0000 | 0.1210 | 0.2420 | 0.3630 | 0.4840 |
| MCTS | 0.0000 | 0.1398 | 0.2795 | 0.4193 | 0.5590 |
| LLM | 0.0160 | 0.1619 | 0.3079 | 0.4538 | 0.5998 |
| LLM-MCTS | 0.0000 | 0.1151 | 0.2302 | 0.3454 | 0.4605 |
| OPRO | 0.0000 | 0.1394 | 0.2788 | 0.4181 | 0.5575 |
| SelfAI w/o stopping | 0.0000 | 0.1469 | 0.2937 | 0.4406 | 0.5874 |
| Random-ES | 0.5404 | 0.4808 | 0.4212 | 0.3616 | 0.3021 |
| BS-ES (TPE-ES) | 0.5627 | 0.4883 | 0.4139 | 0.3395 | 0.2652 |
| MCTS-ES | 0.4755 | 0.4407 | 0.4060 | 0.3712 | 0.3365 |
| MCTS-UCT-stop | 0.1086 | 0.2175 | 0.3263 | 0.4352 | 0.5440 |
| LLM-ES | 0.4269 | 0.4320 | 0.4370 | 0.4421 | 0.4471 |
| Rule-SelfAI | 0.2785 | 0.3501 | 0.4217 | 0.4933 | 0.5648 |
| LLM-MCTS-ES | 0.4464 | 0.3949 | 0.3434 | 0.2919 | 0.2404 |
| SelfAI | 0.5679 | 0.5849 | 0.6019 | 0.6188 | 0.6358 |
Q2. Generalization and intended scope. Generalizing the findings from our controlled benchmark to open-ended scientific discovery depends on three conditions: whether the methods follow a common experimental protocol, whether the outcomes of candidate experiments reliably reflect the final objective, and whether the same code-generation procedure is used when comparing search and stopping strategies. Our current evidence shows that SelfAI’s exploration and stopping mechanisms can be applied to open-ended discovery tasks with AIDE and ML-Master. When these three conditions are met in open-ended agents, we expect SelfAI’s gains in experiment selection and stopping efficiency to carry over to open-ended discovery tasks.
Intended scope. SelfAI is intended for experimental processes with four properties. Each experiment is costly, outcomes can be compared using an evaluation metric, the number of experiments is limited by a budget, and the task can be expressed as a specification containing an objective, admissible experimental choices, evaluation criteria, and a budget. Under these properties, decisions based on the accumulated trajectory about which experiment to run next and whether to continue determine how the budget is allocated and what is ultimately discovered. In open-ended scientific discovery, where a native agent generates and executes the candidates, we expect SelfAI’s gains to carry over when the resulting experimental trajectory is sufficiently rich and reliable—that is, when the native agent produces enough runnable candidates and their validation outcomes reliably reflect the final objective. Such a trajectory provides a reliable evidence base for trajectory analysis and stopping decisions.
Q3. Single-agent baseline, anytime metrics, and other stopping criteria. We thank the reviewer for these questions. SelfAI keeps its advantage in all three comparisons.
Single-agent baseline. A single agent with the same instructions, history and backbone does not achieve similar results (setup and Table D in our response to Reviewer 1, Q3).
Standard anytime metrics. At a fixed trial budget equal to SelfAI's trial count on each problem, every full-budget baseline has a lower hit rate and Best Norm than SelfAI. Matching the patience of the ES baselines to SelfAI's mean trial count leads to the same conclusion (Table C in our response to Reviewer 2, Q3).
Table E. Full-budget baselines cut at SelfAI's trial count on each problem (12.92 trials on average).
| Method | Score | t_best | Hit rate | Best Norm |
|---|---|---|---|---|
| BS (TPE) | 0.3463 | 0.8368 | 0.3077 | 0.9902 |
| Random | 0.4439 | 0.6231 | 0.5077 | 0.9778 |
| SMAC3 | 0.4786 | 0.5681 | 0.6154 | 0.9894 |
| SHA | 0.4137 | 0.6663 | 0.3846 | 0.9124 |
| MCTS | 0.4760 | 0.5748 | 0.6154 | 0.9854 |
| LLM-MCTS | 0.3842 | 0.7403 | 0.3846 | 0.9448 |
| OPRO | 0.4369 | 0.6472 | 0.5385 | 0.9908 |
| LLM | 0.4846 | 0.5844 | 0.5385 | 0.9837 |
| SelfAI | 0.6019 | 0.3642 | 0.9231 | 0.9981 |
Different stopping criteria. The baselines in Table 3 stop by patience (Random-ES, BS-ES, MCTS-ES, LLM-MCTS-ES), by a UCT-based criterion (MCTS-UCT-stop) or by an LLM stopping judgment (LLM-ES), and SelfAI has the highest Score among them. On the same SelfAI search policy, its stopping judgment (Score 0.6019, 12.92 trials) also outperforms patience stopping (0.4217 and 27.15 trials for Rule-SelfAI) and no stopping (0.2937, 35.92 trials).
We thank the reviewer for pressing on what SelfAI adds and on whether the gains come from prompt detail. We further checked all runs behind the tables, and as a result, corrected the setup of several baseline runs on four problems (GraphSAGE, LSTM sentiment analysis, Boston and LCBench) and of two ablation runs. We also corrected the trial counts for several runs in which the same configuration was counted more than once. We replace the corresponding values in Tables 2–5 and the supplementary tables of the revised paper. These corrections do not change the paper’s main conclusions.
Q1. Limitations of AIDE and AIRA that SelfAI addresses and capabilities beyond structured prompts. We thank the reviewer for this question.
Limitations of AIDE and AIRA that SelfAI addresses. Both use feedback inside a fixed search procedure. A hand-designed rule picks the next node (greedy selection in AIDE, and greedy, MCTS or evolutionary selection in AIRA), the LLM acts as an operator that drafts, debugs or improves that node, the search runs until a fixed step or time budget, and the agent is judged by the quality of the final solution. SelfAI addresses these three limitations.
Capabilities beyond combining these functions through structured prompts. In addition to combining these functions through structured prompts, the framework offers other capabilities. The User Agent translates natural-language goals, ideas and constraints into task specifications for users to review and modify before execution. The Experiment Manager runs experiments, handling environment preparation, resource coordination, progress tracking, checkpoint-based recovery and failure reporting, and adds each result to the experimental history (Sect. 3). Furthermore, the framework operates independently of the native agent and runs inside the workflows of AIDE and ML-Master (Supplementary B.7).
W2-1. Prompt detail or framework? We thank the reviewer for this comment. As the reviewer notes, the LLM-ES prompt is much shorter than SelfAI's stopping prompt. We have corrected its description in the Supplementary and Table 4. LLM-ES has the same inputs and stop-or-continue action as SelfAI, but not SelfAI's analysis and planning context or its stopping criteria. LLM-ES+ receives SelfAI's full three-criterion stopping prompt and reaches a hit rate of 0.6923, vs. 0.9231 for SelfAI at a similar number of trials (13.46 vs. 12.92, Table H). LLM-MCTS + SelfAI stop uses the same stopping prompt and reaches 0.6154 (Table 5), and the LLM baseline receives SelfAI's planning rules verbatim yet needs 33.38 trials, vs. 12.92 for SelfAI.
Table H. Baselines given parts of SelfAI's prompts (mean over 13 problems, values from Tables 4, 5 and F). TA denotes trajectory analysis, PR SelfAI's planning rules and SP SelfAI's stopping prompt.