- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether AI agents can perform open-ended scientific discovery, where no explicit objective metric or reward signal is available. The authors extend the Station multi-agent research environment with two mechanisms: a Supervisor agent that encourages research persistence and prevents premature pivots, and Meta Reflection that periodically prompts agents to reassess the scientific value of their directions.
The authors evaluate the system on five open-ended research tasks. Three are rediscovery tasks constructed from recent ICLR oral papers, where the original findings are hidden from agents and used as evaluation targets. Two are exploratory tasks without oracle papers, where findings are assessed by human evaluation and external validation. The proposed Station system recovers 62.7% of predefined sub-discoveries on average, substantially outperforming Codex Multiagent-v2 and AI Scientist-v2 baselines. Additional ablations and behavioral analyses suggest that Supervisor and Meta Reflection improve research persistence and exploration quality.
The paper argues that suitable environment design can enable AI agents to autonomously make meaningful progress in open-ended scientific discovery.
Critical strengths
-
The paper addresses an important and timely question: whether AI agents can move beyond metric-driven optimization toward more open-ended scientific research. This is a fundamental challenge for autonomous research systems, and the evaluation framework is a useful step toward studying it.
-
The authors provide extensive empirical analysis beyond final performance numbers, including ablations, behavioral analysis, and released research trajectories. The analysis of how agents change research directions and respond to reflection provides useful insight into autonomous research behavior.
Critical weaknesses
-
The evidence for genuine open-ended scientific discovery remains limited. Although the paper includes exploratory tasks without oracle papers, this evaluation only covers two tasks with a single run each. Given the stochastic nature of scientific exploration, it is unclear whether the reported discoveries are robust and representative of autonomous scientific discovery more broadly. More extensive evaluation across diverse scientific domains and multiple independent trials would be needed to support the paper's broader claims.
-
The ablation analysis of the proposed mechanisms is incomplete. The paper only evaluates the combined removal of Supervisor and Meta Reflection, and this ablation is conducted only on the emergent-planning task. Since Supervisor and Meta Reflection address different aspects of the research process, separate ablations are needed to understand their individual contributions. Moreover, evaluating the ablation on additional tasks would be important to determine whether the observed improvements generalize beyond a single benchmark.
Questions for authors
-
Station uses a heterogeneous population of frontier models (GPT-5.5, Claude Opus 4.8, and Gemini 3.1 Pro), while the baselines use different model compositions and workflows. Can the authors provide controlled comparisons to disentangle the benefit of heterogeneous multi-model collaboration from the contribution of the Station environment and its proposed mechanisms?
-
Station maintains six heterogeneous frontier-model agents, yet the reported output token usage in Table 2 does not appear substantially larger than some single-workflow baselines. Could the authors clarify how output tokens are aggregated across multiple agents and explain why token usage does not scale proportionally with the number of agents? In particular, does this accounting include all agent generations, spawned agents, reflection steps, and auxiliary model calls?
4 comments
Official Comment by Authors (Part 2)
Response to Weakness 2: Insufficient ablation
To isolate the contributions of the Supervisor and Meta Reflection, we conducted separate ablations on the emergent-planning task, removing one mechanism at a time and running each configuration for 100 ticks. The results are as follows:
| Method | Representation Probing (%) | Planning & Features (%) | Causal Intervention (%) | Overall Score (%) |
|---|---|---|---|---|
| Full Station | 99.07 ± 0.93 | 52.78 ± 14.70 | 62.96 ± 3.70 | 72.39 ± 4.96 |
| Without Meta Reflection | 99.07 ± 0.93 | 41.67 ± 8.33 | 61.73 ± 2.47 | 68.01 ± 3.31 |
| Without Supervisor | 88.89 ± 8.49 | 43.52 ± 7.41 | 60.49 ± 6.17 | 64.65 ± 5.62 |
| Without Both | 87.04 ± 3.34 | 40.74 ± 6.07 | 41.98 ± 10.98 | 57.91 ± 3.52 |
Uncertainties in the tables are standard errors (SE).
We observe that removing the Supervisor reduced performance by 7.74 percentage points, whereas removing Meta Reflection reduced performance by 4.38 percentage points. These results indicate that both mechanisms contribute positively to performance. Further qualitative review shows that, without Meta Reflection, agents are more likely to focus on local problems that may be of limited interest to external researchers, such as how a specific gate in an RNN controls memory, which is less related to planning. Removing the Supervisor leads to more premature pivots; for example, agents may fail to identify how planning is represented in the task, which requires sustained exploration.
We will include the above additional ablation in the revised version of the paper. We agree that further ablations on the remaining tasks would help us better understand the individual contributions of each mechanism. However, because the main goal of this paper is to empirically evaluate the feasibility of open-ended autonomous discovery, we prioritized using our limited budget for baseline comparisons and broader task coverage.
Station uses a heterogeneous population of frontier models (GPT-5.5, Claude Opus 4.8, and Gemini 3.1 Pro), while the baselines use different model compositions and workflows. Can the authors provide controlled comparisons to disentangle the benefit of heterogeneous multi-model collaboration from the contribution of the Station environment and its proposed mechanisms?
To isolate the contribution of model diversity, we evaluated Station using only Gemini agents on the emergent planning task, with three seeds and 100 ticks, and compared it with AI Scientist-v2 using only Gemini agents and with Station using its default mixture of models. The results are as follows:
| Method | Representation Probing (%) | Planning & Features (%) | Causal Intervention (%) | Overall Score (%) |
|---|---|---|---|---|
| AI Scientist-v2 (Gemini) | 41.67 ± 22.05 | 8.33 ± 8.33 | 0.00 ± 0.00 | 18.18 ± 9.09 |
| Station with only Gemini agents | 87.04 ± 6.67 | 46.06 ± 9.69 | 52.16 ± 8.56 | 62.63 ± 6.73 |
| Station with the default model mixture | 99.07 ± 0.93 | 52.78 ± 14.70 | 62.96 ± 3.70 | 72.39 ± 4.96 |
We observe that, after controlling for the model family, Station still substantially outperforms AI Scientist-v2 on the emergent planning task, with a gap of 44.5 percentage points. This suggests that the model mixture alone cannot explain the performance gap. Nonetheless, the default model mixture provides an additional gain of 9.8 percentage points, suggesting that model diversity can further benefit autonomous open-ended discovery, consistent with the intuition illustrated in Fig. 1 that diverse models support broader exploration.
The additional experiment described above will be included in the revised paper, allowing readers to better assess the contributions of model diversity and environment design.
Official Comment by Authors (Part 3)
Station maintains six heterogeneous frontier-model agents, yet the reported output token usage in Table 2 does not appear substantially larger than some single-workflow baselines. Could the authors clarify how output tokens are aggregated across multiple agents and explain why token usage does not scale proportionally with the number of agents? In particular, does this accounting include all agent generations, spawned agents, reflection steps, and auxiliary model calls?
We would like to clarify that both Station and the baselines, Codex Multiagent-v2 and AI Scientist-v2, are multi-agent systems. Station uses six concurrent agents, Codex Multiagent-v2 uses 32 sub-agents under a root agent, and AI Scientist-v2 uses eight concurrent agents during tree expansion. Although the baselines use centralized orchestration, they still employ multiple agents for parallel exploration. Therefore, the token usage of Station and the baselines generally scales with the number of concurrent agents deployed, and the baselines’ total token usage can be similar to or exceed Station’s under comparable runtime budgets.
For token accounting, the reported output-token count is the total usage of all agents across the entire system, for both Station and the baselines, as reported by the respective API providers. This includes all of the cases you mentioned above.
We thank you again for your valuable feedback. We hope that our response and additional experiments clarify the paper’s methodology and contributions. We would be grateful if you would consider this additional evidence in your final assessment. We welcome any further questions.
[1] Kirgis, P., et al. (2026). Can AI agents conduct open-ended AI research? Early evidence from two case studies. arXiv preprint arXiv:2607.27191. https://arxiv.org/abs/2607.27191
Dear Reviewer RoFx,
Thank you again for taking time to review our submission. As the rebuttal period is nearing its end with only three days remaining, we wanted to kindly follow up and ask whether our clarifications have adequately addressed your concerns.
Thank you again for your time and consideration!
Official Comment by Authors (Part 1)
We thank the reviewer for their constructive feedback and detailed comments. We address the reviewer's concerns below:
Response to Weakness 1: Limited evaluation
To broaden the scope of the evaluation, we added one rediscovery task based on a concurrent paper that also investigates open-ended discovery [1]. In this task, agents were asked to investigate the structure and controllability of LLM personas, including whether personality directions in weight space are composable, transferable, and shared across models. The original paper was withheld from the agents, and web access was disabled. Since the original paper was released after the models’ knowledge cutoff dates, achieving a high score required the agents to make discoveries that were novel relative to their pretraining data, similar to exploratory tasks.
We used the same evaluation format as for our other rediscovery tasks, assessing ten findings in three categories. Trait control and composition covers five findings about how weight changes control personas, and whether different trait directions can be combined or reversed. Capability trade-offs covers two findings about achieving useful persona control while preserving general capabilities, and how stronger interventions can impair them. Robustness covers three findings about reproducing control across base models and data-generating teachers, and whether the results still hold with a simpler weight adjustment. In addition to the existing baselines, we also evaluated the paper generated in [1] using OpenClaw (Claude Opus 4.8) on the same task, applying our evaluation system for comparison.
The results are as follows:
The research reports and dialogue logs from the Station runs have been uploaded to the repository for transparency.
We observe that Station recovers more findings than the baselines, with contributions across all three categories. Our review of Station’s results shows examples of building on initial observations through further investigation: agents initially found that combining different trait adjustments in weight space did not produce the intended mixture of traits in the model’s behavior. Based on this finding, they separated out changes shared by the adjustments and tuned how the adjustments were combined to reduce interference between traits, making their combined effects more predictable. In contrast, Kirgis et al. [1] report that their OpenClaw system committed prematurely to one approach and struggled to turn initial observations into meaningful downstream findings. We observed similar difficulties in the baselines. As such, this task provides further evidence that a suitable research environment can help agents build on intermediate findings to make deeper discoveries in open-ended research. We will add this experiment and its details to the revised version of the paper.
Finally, we would also like to point out that open-ended autonomous discovery remains considerably expensive. With the additional experiment above, we now have four rediscovery tasks, each with three seeds and one Station run plus four baseline runs, for a total of 4 × 5 × 3 = 60 runs, in addition to the two exploratory tasks. The existing evaluation had already consumed over 760 million output tokens and 3,400 hours of cumulative experiment time, as reported in Appendix E.3. In comparison, similar work, such as [1], reports an evaluation based on only two tasks. Although we agree that adding more tasks would strengthen the evaluation, doing so is currently beyond our computational budget, and we hope the reviewer understands.