acceptodds
Under review as a conference paper at ICLR 2027

Can AI Agents Make Open-Ended Scientific Discovery? Evidence from Station

Abstract

Recent AI systems have made rapid progress in scientific discovery when given well-defined metrics, but whether they can autonomously undertake open-ended scientific discovery remains unclear. We investigate AI’s ability to tackle open-ended tasks in Station, an open-world environment in which multiple agents simulate a scientific ecosystem. To tackle challenges specific to open-ended tasks, we propose augmenting Station with two mechanisms: a Supervisor mechanism and periodic Meta Reflection, which encourage persistent exploration even when intermediate metrics are lacking. We construct open-ended tasks from three recent oral papers presented at ICLR. We give agents the main research question studied in each paper while withholding the paper’s results and disabling web access. We then measure how many of the original findings—partitioned into individual criteria—agents rediscover. We find that Station rediscovers 62.7% of the criteria on average, compared with 15.4% for Codex Multiagent-v2 and 14.4-20.6% for AI Scientist-v2. Ablation and behavioral analyses indicate that adding the two mechanisms together improves research coverage and continuity. We further evaluate Station on two open-ended tasks without oracle papers and find that some of the discoveries made by the agents closely match discoveries reported by researchers after the knowledge cutoff date. Together, these results indicate that a suitable environment can enable agents to autonomously make meaningful progress in open-ended scientific discovery.

Primary area:
foundation or frontier models, including LLMs
Keywords:
AI Scientist, Autonomous Scientific Discovery, AI Agents, Multi-Agent Systems, AI for Science

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RoFxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether AI agents can perform open-ended scientific discovery, where no explicit objective metric or reward signal is available. The authors extend the Station multi-agent research environment with two mechanisms: a Supervisor agent that encourages research persistence and prevents premature pivots, and Meta Reflection that periodically prompts agents to reassess the scientific value of their directions.

The authors evaluate the system on five open-ended research tasks. Three are rediscovery tasks constructed from recent ICLR oral papers, where the original findings are hidden from agents and used as evaluation targets. Two are exploratory tasks without oracle papers, where findings are assessed by human evaluation and external validation. The proposed Station system recovers 62.7% of predefined sub-discoveries on average, substantially outperforming Codex Multiagent-v2 and AI Scientist-v2 baselines. Additional ablations and behavioral analyses suggest that Supervisor and Meta Reflection improve research persistence and exploration quality.

The paper argues that suitable environment design can enable AI agents to autonomously make meaningful progress in open-ended scientific discovery.

Critical strengths
  1. The paper addresses an important and timely question: whether AI agents can move beyond metric-driven optimization toward more open-ended scientific research. This is a fundamental challenge for autonomous research systems, and the evaluation framework is a useful step toward studying it.

  2. The authors provide extensive empirical analysis beyond final performance numbers, including ablations, behavioral analysis, and released research trajectories. The analysis of how agents change research directions and respond to reflection provides useful insight into autonomous research behavior.

Critical weaknesses
  1. The evidence for genuine open-ended scientific discovery remains limited. Although the paper includes exploratory tasks without oracle papers, this evaluation only covers two tasks with a single run each. Given the stochastic nature of scientific exploration, it is unclear whether the reported discoveries are robust and representative of autonomous scientific discovery more broadly. More extensive evaluation across diverse scientific domains and multiple independent trials would be needed to support the paper's broader claims.

  2. The ablation analysis of the proposed mechanisms is incomplete. The paper only evaluates the combined removal of Supervisor and Meta Reflection, and this ablation is conducted only on the emergent-planning task. Since Supervisor and Meta Reflection address different aspects of the research process, separate ablations are needed to understand their individual contributions. Moreover, evaluating the ablation on additional tasks would be important to determine whether the observed improvements generalize beyond a single benchmark.

Questions for authors
  1. Station uses a heterogeneous population of frontier models (GPT-5.5, Claude Opus 4.8, and Gemini 3.1 Pro), while the baselines use different model compositions and workflows. Can the authors provide controlled comparisons to disentangle the benefit of heterogeneous multi-model collaboration from the contribution of the Station environment and its proposed mechanisms?

  2. Station maintains six heterogeneous frontier-model agents, yet the reported output token usage in Table 2 does not appear substantially larger than some single-workflow baselines. Could the authors clarify how output tokens are aggregated across multiple agents and explain why token usage does not scale proportionally with the number of agents? In particular, does this accounting include all agent generations, spawned agents, reflection steps, and auxiliary model calls?

4 comments
AuthorsOpenReview ↗

Official Comment by Authors (Part 1)

We thank the reviewer for their constructive feedback and detailed comments. We address the reviewer's concerns below:

Response to Weakness 1: Limited evaluation

To broaden the scope of the evaluation, we added one rediscovery task based on a concurrent paper that also investigates open-ended discovery [1]. In this task, agents were asked to investigate the structure and controllability of LLM personas, including whether personality directions in weight space are composable, transferable, and shared across models. The original paper was withheld from the agents, and web access was disabled. Since the original paper was released after the models’ knowledge cutoff dates, achieving a high score required the agents to make discoveries that were novel relative to their pretraining data, similar to exploratory tasks.

We used the same evaluation format as for our other rediscovery tasks, assessing ten findings in three categories. Trait control and composition covers five findings about how weight changes control personas, and whether different trait directions can be combined or reversed. Capability trade-offs covers two findings about achieving useful persona control while preserving general capabilities, and how stronger interventions can impair them. Robustness covers three findings about reproducing control across base models and data-generating teachers, and whether the results still hold with a simpler weight adjustment. In addition to the existing baselines, we also evaluated the paper generated in [1] using OpenClaw (Claude Opus 4.8) on the same task, applying our evaluation system for comparison.

The results are as follows:

MethodPersona Control (%)Capability Trade-off (%)Robustness (%)Overall Score (%)
Station (Seed 1)71.1172.2237.0461.11
Station (Seed 2)80.0066.6733.3363.33
Station (Seed 3)71.1138.8933.3353.33
Station (Overall Mean)74.0759.2634.5759.26
OpenClaw (Claude Opus 4.8)40.000.0033.3330.00
Codex Multiagent-v2 (GPT)40.0050.0033.3340.00
AI Scientist-v2 (GPT)20.0050.000.0020.00
AI Scientist-v2 (Gemini)40.0066.670.0033.33
AI Scientist-v2 (Claude)80.000.0033.3350.00

The research reports and dialogue logs from the Station runs have been uploaded to the repository for transparency.

We observe that Station recovers more findings than the baselines, with contributions across all three categories. Our review of Station’s results shows examples of building on initial observations through further investigation: agents initially found that combining different trait adjustments in weight space did not produce the intended mixture of traits in the model’s behavior. Based on this finding, they separated out changes shared by the adjustments and tuned how the adjustments were combined to reduce interference between traits, making their combined effects more predictable. In contrast, Kirgis et al. [1] report that their OpenClaw system committed prematurely to one approach and struggled to turn initial observations into meaningful downstream findings. We observed similar difficulties in the baselines. As such, this task provides further evidence that a suitable research environment can help agents build on intermediate findings to make deeper discoveries in open-ended research. We will add this experiment and its details to the revised version of the paper.

Finally, we would also like to point out that open-ended autonomous discovery remains considerably expensive. With the additional experiment above, we now have four rediscovery tasks, each with three seeds and one Station run plus four baseline runs, for a total of 4 × 5 × 3 = 60 runs, in addition to the two exploratory tasks. The existing evaluation had already consumed over 760 million output tokens and 3,400 hours of cumulative experiment time, as reported in Appendix E.3. In comparison, similar work, such as [1], reports an evaluation based on only two tasks. Although we agree that adding more tasks would strengthen the evaluation, doing so is currently beyond our computational budget, and we hope the reviewer understands.

AuthorsOpenReview ↗

Official Comment by Authors (Part 2)

Response to Weakness 2: Insufficient ablation

To isolate the contributions of the Supervisor and Meta Reflection, we conducted separate ablations on the emergent-planning task, removing one mechanism at a time and running each configuration for 100 ticks. The results are as follows:

MethodRepresentation Probing (%)Planning & Features (%)Causal Intervention (%)Overall Score (%)
Full Station99.07 ± 0.9352.78 ± 14.7062.96 ± 3.7072.39 ± 4.96
Without Meta Reflection99.07 ± 0.9341.67 ± 8.3361.73 ± 2.4768.01 ± 3.31
Without Supervisor88.89 ± 8.4943.52 ± 7.4160.49 ± 6.1764.65 ± 5.62
Without Both87.04 ± 3.3440.74 ± 6.0741.98 ± 10.9857.91 ± 3.52

Uncertainties in the tables are standard errors (SE).

We observe that removing the Supervisor reduced performance by 7.74 percentage points, whereas removing Meta Reflection reduced performance by 4.38 percentage points. These results indicate that both mechanisms contribute positively to performance. Further qualitative review shows that, without Meta Reflection, agents are more likely to focus on local problems that may be of limited interest to external researchers, such as how a specific gate in an RNN controls memory, which is less related to planning. Removing the Supervisor leads to more premature pivots; for example, agents may fail to identify how planning is represented in the task, which requires sustained exploration.

We will include the above additional ablation in the revised version of the paper. We agree that further ablations on the remaining tasks would help us better understand the individual contributions of each mechanism. However, because the main goal of this paper is to empirically evaluate the feasibility of open-ended autonomous discovery, we prioritized using our limited budget for baseline comparisons and broader task coverage.

Station uses a heterogeneous population of frontier models (GPT-5.5, Claude Opus 4.8, and Gemini 3.1 Pro), while the baselines use different model compositions and workflows. Can the authors provide controlled comparisons to disentangle the benefit of heterogeneous multi-model collaboration from the contribution of the Station environment and its proposed mechanisms?

To isolate the contribution of model diversity, we evaluated Station using only Gemini agents on the emergent planning task, with three seeds and 100 ticks, and compared it with AI Scientist-v2 using only Gemini agents and with Station using its default mixture of models. The results are as follows:

MethodRepresentation Probing (%)Planning & Features (%)Causal Intervention (%)Overall Score (%)
AI Scientist-v2 (Gemini)41.67 ± 22.058.33 ± 8.330.00 ± 0.0018.18 ± 9.09
Station with only Gemini agents87.04 ± 6.6746.06 ± 9.6952.16 ± 8.5662.63 ± 6.73
Station with the default model mixture99.07 ± 0.9352.78 ± 14.7062.96 ± 3.7072.39 ± 4.96

We observe that, after controlling for the model family, Station still substantially outperforms AI Scientist-v2 on the emergent planning task, with a gap of 44.5 percentage points. This suggests that the model mixture alone cannot explain the performance gap. Nonetheless, the default model mixture provides an additional gain of 9.8 percentage points, suggesting that model diversity can further benefit autonomous open-ended discovery, consistent with the intuition illustrated in Fig. 1 that diverse models support broader exploration.

The additional experiment described above will be included in the revised paper, allowing readers to better assess the contributions of model diversity and environment design.

AuthorsOpenReview ↗

Official Comment by Authors (Part 3)

Station maintains six heterogeneous frontier-model agents, yet the reported output token usage in Table 2 does not appear substantially larger than some single-workflow baselines. Could the authors clarify how output tokens are aggregated across multiple agents and explain why token usage does not scale proportionally with the number of agents? In particular, does this accounting include all agent generations, spawned agents, reflection steps, and auxiliary model calls?

We would like to clarify that both Station and the baselines, Codex Multiagent-v2 and AI Scientist-v2, are multi-agent systems. Station uses six concurrent agents, Codex Multiagent-v2 uses 32 sub-agents under a root agent, and AI Scientist-v2 uses eight concurrent agents during tree expansion. Although the baselines use centralized orchestration, they still employ multiple agents for parallel exploration. Therefore, the token usage of Station and the baselines generally scales with the number of concurrent agents deployed, and the baselines’ total token usage can be similar to or exceed Station’s under comparable runtime budgets.

For token accounting, the reported output-token count is the total usage of all agents across the entire system, for both Station and the baselines, as reported by the respective API providers. This includes all of the cases you mentioned above.

We thank you again for your valuable feedback. We hope that our response and additional experiments clarify the paper’s methodology and contributions. We would be grateful if you would consider this additional evidence in your final assessment. We welcome any further questions.

[1] Kirgis, P., et al. (2026). Can AI agents conduct open-ended AI research? Early evidence from two case studies. arXiv preprint arXiv:2607.27191. https://arxiv.org/abs/2607.27191

Dear Reviewer RoFx,

Thank you again for taking time to review our submission. As the rebuttal period is nearing its end with only three days remaining, we wanted to kindly follow up and ask whether our clarifications have adequately addressed your concerns.

Thank you again for your time and consideration!

Reviewer_wCWSOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the capability of AI agents to autonomously perform open-ended scientific discovery in domains that lack well-defined, objective progress metrics. To address the challenges of judgment misalignment and the streetlight effect, the authors adapt Station, a decentralized multi-agent research environment, by introducing two novel mechanisms. The first is a Supervisor mechanism that enforces a minimum commitment period to a proposed research direction, preventing premature pivoting. The second is a periodic Meta Reflection protocol that prompts agents to self-assess their research alignment with human scientific values. The system is evaluated on three rediscovery tasks, where agents must uncover withheld findings from recent ICLR oral papers, and two exploratory tasks without oracle papers. The extended Station successfully recovers over sixty percent of the withheld sub-discoveries, significantly outperforming centralized baselines such as AI Scientist-v2 and Codex Multiagent-v2. Furthermore, on the exploratory tasks, the agents independently discover mechanisms that closely match concurrent human research and propose novel hypotheses, demonstrating the potential of suitably designed environments to enable meaningful progress in open-ended scientific discovery.

Critical strengths

The primary strength of this work lies in its focus on a crucial and highly difficult problem, specifically transitioning AI scientific discovery from tightly constrained, metric-driven tasks to open-ended, subjective domains. The authors propose highly creative and innovative environmental interventions rather than merely relying on prompt engineering. By structurally altering the incentives and lifecycle of the environment through the Supervisor and Meta Reflection mechanisms, the system effectively forces sustained exploration and periodic realignment with human research tastes. Additionally, the paper demonstrates a rigorous and transparent evaluation methodology. The decomposition of oracle papers into granular sub-discoveries provides a measurable metric, and the blinded expert assessment, which shows a remarkably high agreement rate between the automated evaluator and a human author, strongly validates the automated rubric. The empirical results on the exploratory tasks are particularly impressive, as the system's ability to independently discover actionable insights that match concurrent human work highlights its practical scientific utility. Finally, the authors exhibit commendable candor regarding the remaining gaps between AI and human researchers, explicitly acknowledging that agents still struggle with deep mechanistic explanations and occasionally waste effort on uninteresting directions.

Critical weaknesses

Despite its contributions, the paper presents a few notable weaknesses, primarily concerning the inherent contradiction in evaluating open-ended discovery through rediscovery tasks. Measuring a system's ability to make open-ended discoveries by how well it finds a hidden, pre-defined human truth is paradoxical, as an agent that discovers a novel but completely different valid mechanism would be penalized by the metric. Furthermore, the massive performance gap between Station and the centralized baselines, which perform exceptionally poorly, raises questions about experimental fairness and whether the baselines are fundamentally incompatible with the multi-model, long-horizon setup used in this study. A deeper analysis into why the baselines fail so catastrophically would strengthen the paper's claims. There are also significant concerns regarding compute and scalability, as the system requires a massive computational budget of thousands of GPU hours and millions of dollars in API costs to rediscover a single paper's findings, limiting its practical utility for everyday researchers. Additionally, it is difficult to disentangle how much of the success is due to the brilliant environmental design versus the simple application of a highly capable, expensive cocktail of frontier models. Lastly, while the blinded validation is reassuring, the initial creation of the sub-discovery rubrics still relies on the authors' subjective interpretation of the oracle papers' main findings, which could introduce bias into the evaluation framework.

Questions for authors

please see weakness.

3 comments
AuthorsOpenReview ↗

Official Comment by Authors (Part 1)

We thank the reviewer for their constructive feedback and detailed comments. We address the reviewer's concerns below:

Measuring a system's ability to make open-ended discoveries by how well it finds a hidden, pre-defined human truth is paradoxical, as an agent that discovers a novel but completely different valid mechanism would be penalized by the metric.

We agree with you that there is a possibility that agents may discover novel and significant findings that are not included in the oracle paper. However, for all rediscovery tasks, we manually screened the agents’ findings beyond those that overlapped with the oracle paper and assessed their significance. We found that these findings were mostly local and relatively minor, so we did not include them in the paper. We conjecture that, for the three selected tasks, all of which are based on oral papers, the original authors had already conducted sufficiently comprehensive investigations that left little room for overlooked yet significant findings. Nonetheless, all agent findings have been uploaded to the public repository so that they can be independently evaluated if needed.

In contrast, the exploratory tasks we present do not have oracle papers. The agents made discoveries that coincided with findings from concurrent work published after their knowledge cutoffs. This provides evidence that agents can make discoveries on open-ended tasks, although such discoveries are difficult to quantify without an oracle paper. There is therefore a trade-off between evaluating a system with and without an oracle paper, and we include both settings in our paper to provide a more comprehensive evaluation.

the massive performance gap between Station and the centralized baselines, which perform exceptionally poorly, raises questions about experimental fairness and whether the baselines are fundamentally incompatible with the multi-model, long-horizon setup used in this study.

We would like to clarify that we did not run the baselines with a mixture of models because their original implementations do not support such a mixture. For example, for AI Scientist-v2, we ran the baseline separately with each model family used in Station, resulting in 3 seeds × 3 model families = 9 runs for each rediscovery task. We also conducted an additional experiment comparing Station using only Gemini agents with AI Scientist-v2 using only Gemini agents to control for the model family; see our response below.

Both OpenAI’s Codex Multiagent-v2 and AI Scientist-v2 were designed for long-horizon investigations. For example, Codex Multiagent-v2 was used in OpenAI’s proof of the Cycle Double Cover Conjecture [1]. All baselines were implemented using their official repositories. We report token usage and control for cumulative experiment time, as discussed in Appendix E.3. Therefore, we believe that our baseline comparisons are fair.

We believe that the large performance gap between Station and the baselines is mainly attributable to the baselines’ centralized design. As discussed in Sec. 6, our qualitative analysis suggests that baseline agents often focus on narrower subproblems and struggle to build on intermediate findings to make deeper discoveries. This difficulty is also reported independently by Kirgis et al. [2], whose OpenClaw system committed prematurely to one approach and struggled to develop initial observations into meaningful downstream findings. To investigate this further, we added a new experiment evaluating Station and the baselines on the persona task from [2]. We observed similar difficulties in the baselines, while Station built on earlier findings and achieved a higher overall score (see details below). These observations are consistent with our hypothesis in Fig. 1: centralized orchestration can make exploration overly dependent on the direction selected by a single controller, while a decentralized environment allows agents to pursue different directions in parallel and build on one another’s findings.

the system requires a massive computational budget of thousands of GPU hours and millions of dollars in API costs to rediscover a single paper's findings, limiting its practical utility for everyday researchers.

As discussed in Appendix E.3, a Station run requires approximately 5.8–11.0 million output tokens. On average, this translates to approximately USD 500 in API costs per run. Compute costs depend on the nature of the task, particularly whether it requires substantial GPU usage. For the task of emergent planning, a typical run requires around 44 GPU-hours, corresponding to around USD 88 at current rental rates. For typical researcher use, where multiple random seeds are unnecessary, we believe that this remains affordable. We also expect these costs to decrease over time, given the recent downward trend in frontier-model pricing. We will include a cost estimate in the appendix to clarify this point.

AuthorsOpenReview ↗

Official Comment by Authors (Part 2)

it is difficult to disentangle how much of the success is due to the brilliant environmental design versus the simple application of a highly capable, expensive cocktail of frontier models.

To isolate the contribution of model diversity, we added an experiment in which Station using only Gemini agents on the emergent planning task, with three seeds and 100 ticks, and compared it with AI Scientist-v2 using only Gemini agents and with Station using its default mixture of models. The results are as follows:

MethodRepresentation Probing (%)Planning & Features (%)Causal Intervention (%)Overall Score (%)
AI Scientist-v2 (Gemini)41.67 ± 22.058.33 ± 8.330.00 ± 0.0018.18 ± 9.09
Station with only Gemini agents87.04 ± 6.6746.06 ± 9.6952.16 ± 8.5662.63 ± 6.73
Station with the default model mixture99.07 ± 0.9352.78 ± 14.7062.96 ± 3.7072.39 ± 4.96

Uncertainties in the tables are standard errors (SE).

We observe that, after controlling for the model family, Station still substantially outperforms AI Scientist-v2 on the emergent planning task, with a gap of 44.5 percentage points. This suggests that the model mixture alone cannot explain the performance gap. Nonetheless, the default model mixture provides an additional gain of 9.8 percentage points, suggesting that model diversity can further benefit autonomous open-ended discovery, consistent with the intuition illustrated in Fig. 1 that diverse models support broader exploration.

The additional experiment described above will be included in the revised paper, allowing readers to better assess the contributions of model diversity and environment design.

While the blinded validation is reassuring, the initial creation of the sub-discovery rubrics still relies on the authors' subjective interpretation of the oracle papers' main findings, which could introduce bias into the evaluation framework.

We agree with you that the rubric may introduce subjective preferences. We mitigate this concern by defining the criteria before running any experiments and using the same criteria for both Station and the baselines. We also selected only findings presented in the main paper, rather than in the appendix, because we believe that the oracle paper’s authors likely considered them more important. Finally, all agent findings have been uploaded to our public repository, allowing them to be independently re-evaluated using alternative criteria that other researchers may wish to apply.

We thank you again for your valuable feedback. We hope that our response and additional experiments clarify the paper’s methodology and contributions. We would be grateful if you would consider this additional evidence in your final assessment. We welcome any further questions.

[1] OpenAI. “Prompt Used for ‘A Proof of the Cycle Double Cover Conjecture.’” 2026. https://cdn.openai.com/pdf/04d1d1e4-bc75-476a-97cf-49055cd98d31/cdc_prompt.pdf

[2] Kirgis, P., et al. (2026). Can AI agents conduct open-ended AI research? Early evidence from two case studies. arXiv preprint arXiv:2607.27191. https://arxiv.org/abs/2607.27191

Dear Reviewer wCWS,

Thank you again for taking time to review our submission. As the rebuttal period is nearing its end with only three days remaining, we wanted to kindly follow up and ask whether our clarifications have adequately addressed your concerns.

Thank you again for your time and consideration!

Reviewer_nxPhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether multi-agent AI systems can do open-ended scientific discovery when no objective metric guides exploration. It extends Station, an existing decentralized multi-agent research environment, with a Supervisor agent that blocks early pivots, periodic Meta Reflection, and a Consolidation Agent that writes the final report. On three rediscovery tasks built from ICLR oral papers, agents receive the research question and experimental setup but not the findings, and an LLM judge checks how many of about ten predefined sub-discoveries the final report recovers. Station recovers 62.7% on average, compared with 14.4–20.6% for AI Scientist-v2 and Codex Multiagent-v2 under matched experiment time. An ablation and behavioral analyses suggest the two mechanisms make research more persistent. On two exploratory tasks without oracle papers, agents produce findings that match concurrent human work, plus one claimed novel result on subliminal learning.

Critical strengths
  1. Concrete protocol for evaluating open-ended discovery. Giving agents an oracle paper's research question and setup while withholding its findings, then scoring against rubrics fixed before any runs, makes open-ended research measurable without reducing it to a single metric.

  2. Large, consistent gains over strong baselines. Station beats AI Scientist-v2 and Codex Multiagent-v2 on all three tasks, most clearly on deeper findings such as planning features, causal interventions and functional consequences.

Critical weaknesses
  1. The gains cannot be attributed to the proposed mechanisms. Removing Supervisor and Meta Reflection together, on one task only, still leaves Station at 57.9% against at most 19.2% for the baselines, so most of the gap exists before the new mechanisms are added and is never explained. Beyond the original Station, the system also differs from the baselines in its mix of model families and in a separate consolidation agent that writes its report, so better report writing may explain part of the gain.

  2. The evaluation is narrow. There are only three rediscovery tasks with three runs each, and the direct-prompting control cited against training-data exposure is never reported. The judge was validated on only 6 reports from one task, and the exploratory findings come from single runs screened by hand.

Questions for authors
  1. What explains most of the gap over the baselines? For example, how do the baselines score if their research records are written up by the same Consolidation Agent?
  2. Can you report the direct-prompting results mentioned in the limitations, to show how much of the oracle findings the models recover without running any research?
5 comments
AuthorsOpenReview ↗

Official Comment by Authors (Part 1)

We thank the reviewer for their constructive feedback and detailed comments. We address the reviewer's concerns below:

Response to Weakness 1: The gains cannot be attributed to the proposed mechanisms.

You are right that the gap between Station and the baselines is already large before adding the Supervisor and Meta Reflection. We explain the sources of this gap below.

As discussed in Sec. 6, our qualitative analysis suggests that baseline agents often focus on narrower subproblems and struggle to build on intermediate findings to make deeper discoveries. This difficulty is also reported independently by Kirgis et al. [1], whose OpenClaw system committed prematurely to one approach and struggled to develop initial observations into meaningful downstream findings. To investigate this further, we added a new experiment evaluating Station and the baselines on the persona task from [1]. We observed similar difficulties in the baselines, while Station built on earlier findings and achieved a higher overall score (see details below). These observations are consistent with our hypothesis in Fig. 1: centralized orchestration can make exploration overly dependent on the direction selected by a single controller, while a decentralized environment allows agents to pursue different directions in parallel and build on one another’s findings. We believe this difference is a major contributor to the gap between Station and the baselines.

We would also like to clarify that this work is both an empirical study of Station applied to open-ended scientific discovery and an extension of Station with additional mechanisms. The original Station was designed and evaluated on scorable tasks with explicit evaluation scores, and whether such a system can be applied to open-ended discovery is an important research question in its own right. Similarly, Kirgis et al. [1] investigate the capabilities and limitations of existing systems in open-ended autonomous research. Our proposed mechanisms, including Meta Reflection and the Supervisor, further adapt Station to open-ended tasks, and we show that they improve performance beyond the existing system.

Beyond the original Station, the system also differs from the baselines in its mix of model families and in a separate consolidation agent that writes its report, so better report writing may explain part of the gain.

Because each baseline supports only a single model family for each task, we did not use a mix of model families for the baselines, as doing so would require modifications beyond the scope of the original work. Instead, we ran baselines with all three model families used in Station and reported the results for each configuration. However, we agree that using the same model would enable a more consistent comparison. We therefore added an experiment in which we evaluated Station using only Gemini agents on the emergent planning task, with three seeds and 100 ticks, and compared it with AI Scientist-v2 using only Gemini agents and with Station using its default mixture of models. The results are as follows:

MethodRepresentation Probing (%)Planning & Features (%)Causal Intervention (%)Overall Score (%)
AI Scientist-v2 (Gemini)41.67 ± 22.058.33 ± 8.330.00 ± 0.0018.18 ± 9.09
Station with only Gemini agents87.04 ± 6.6746.06 ± 9.6952.16 ± 8.5662.63 ± 6.73
Station with the default model mixture99.07 ± 0.9352.78 ± 14.7062.96 ± 3.7072.39 ± 4.96

Uncertainties in the tables are standard errors (SE).

We observe that, after controlling for the model family, Station still substantially outperforms AI Scientist-v2 on the emergent planning task, with a gap of 44.5 percentage points. This suggests that the model mixture alone cannot explain the performance gap. Nonetheless, the default model mixture provides an additional gain of 9.8 percentage points, suggesting that model diversity can further benefit autonomous open-ended discovery, consistent with the intuition illustrated in Fig. 1 that diverse models support broader exploration.

AuthorsOpenReview ↗

Official Comment by Authors (Part 2)

As for the contribution of report writing, we conducted a further experiment to isolate its effect by running the same Consolidation Agent on the internal records produced by the baselines. Specifically, we allowed the Consolidation Agent to access all files and results produced by each baseline system, as in Station, and then asked it to write an 8,000-word report using the same prompt as the one used for Station. This provides a fair comparison because the same Consolidation Agent is applied to both methods. The result is as follows:

Emergent Planning in RL Agents

MethodRepresentation Probing (%)Planning & Features (%)Causal Intervention (%)Overall Score (%)
Station (Tick 100)99.07 ± 0.9352.78 ± 14.7062.96 ± 3.7072.39 ± 4.96
Codex Multiagent-v2 (GPT)19.44 ± 10.020.00 ± 0.000.00 ± 0.007.07 ± 3.64
Codex Multiagent-v2 (GPT) + Consolidation11.11 ± 7.350.00 ± 0.000.00 ± 0.004.04 ± 2.67
AI Scientist-v2 (Claude)52.78 ± 2.780.00 ± 0.000.00 ± 0.0019.19 ± 1.01
AI Scientist-v2 (Claude) + Consolidation50.00 ± 0.005.56 ± 5.560.00 ± 0.0020.20 ± 2.02
AI Scientist-v2 (Gemini)41.67 ± 22.058.33 ± 8.330.00 ± 0.0018.18 ± 9.09
AI Scientist-v2 (Gemini) + Consolidation38.89 ± 15.478.33 ± 8.3311.11 ± 11.1120.20 ± 5.35
AI Scientist-v2 (GPT)16.67 ± 4.810.00 ± 0.000.00 ± 0.006.06 ± 1.75
AI Scientist-v2 (GPT) + Consolidation19.44 ± 5.560.00 ± 0.000.00 ± 0.007.07 ± 2.02

Low-rank structure in LLM outputs

MethodSequence-Level Structure (%)Reusable Relations (%)Functional Consequences (%)Overall Score (%)
Station (Tick 100)100.00 ± 0.0092.59 ± 4.0418.52 ± 13.0670.37 ± 5.66
Codex Multiagent-v2 (GPT)88.89 ± 5.5611.11 ± 5.560.00 ± 0.0033.33 ± 0.00
Codex Multiagent-v2 (GPT) + Consolidation100.00 ± 0.000.00 ± 0.000.00 ± 0.0033.33 ± 0.00
AI Scientist-v2 (Claude)86.11 ± 7.3511.11 ± 5.560.00 ± 0.0032.41 ± 3.34
AI Scientist-v2 (Claude) + Consolidation94.44 ± 5.560.00 ± 0.000.00 ± 0.0031.48 ± 1.85
AI Scientist-v2 (Gemini)83.33 ± 8.330.00 ± 0.005.56 ± 5.5629.63 ± 2.45
AI Scientist-v2 (Gemini) + Consolidation91.67 ± 4.810.00 ± 0.008.33 ± 8.3333.33 ± 1.60
AI Scientist-v2 (GPT)97.22 ± 2.7813.89 ± 2.780.00 ± 0.0037.04 ± 0.93
AI Scientist-v2 (GPT) + Consolidation97.22 ± 2.780.00 ± 0.000.00 ± 0.0032.41 ± 0.93

Temporal representations in RNNs

MethodLinear Dynamics (%)Nonlinear Dynamics (%)Spatial–Temporal Tradeoffs (%)Overall Score (%)
Station (Tick 100)51.85 ± 14.8125.93 ± 17.5043.52 ± 16.0640.74 ± 9.82
Codex Multiagent-v2 (GPT)22.22 ± 22.220.00 ± 0.000.00 ± 0.006.67 ± 6.67
Codex Multiagent-v2 (GPT) + Consolidation25.93 ± 16.140.00 ± 0.005.56 ± 5.5610.00 ± 5.09
AI Scientist-v2 (Claude)3.70 ± 3.700.00 ± 0.000.00 ± 0.001.11 ± 1.11
AI Scientist-v2 (Claude) + Consolidation0.00 ± 0.000.00 ± 0.000.00 ± 0.000.00 ± 0.00
AI Scientist-v2 (Gemini)33.33 ± 19.253.70 ± 3.700.00 ± 0.0011.11 ± 6.76
AI Scientist-v2 (Gemini) + Consolidation22.22 ± 16.977.41 ± 7.418.33 ± 8.3312.22 ± 5.88
AI Scientist-v2 (GPT)0.00 ± 0.000.00 ± 0.000.00 ± 0.000.00 ± 0.00
AI Scientist-v2 (GPT) + Consolidation0.00 ± 0.000.00 ± 0.000.00 ± 0.000.00 ± 0.00

We observe no significant change in the baselines’ total scores after adding the Consolidating Agent. Since its primary role is to aggregate existing findings, and the low scores mainly reflect missing findings, we expect that changing how the findings are organized would not materially affect the results.

We will add the above experiment to the revised paper to strengthen our comparison with the baselines. We hope that these additional experiments address your concerns about the baseline comparison.

AuthorsOpenReview ↗

Official Comment by Authors (Part 3)

Response to Weakness 2: The evaluation is narrow.

To broaden the scope of the evaluation, we added one rediscovery task based on a concurrent paper that also investigates open-ended discovery [1]. In this task, agents were asked to investigate the structure and controllability of LLM personas, including whether personality directions in weight space are composable, transferable, and shared across models. The original paper was withheld from the agents, and web access was disabled.

We used the same evaluation format as for our other rediscovery tasks, assessing ten findings in three categories. Trait control and composition covers five findings about how weight changes control personas, and whether different trait directions can be combined or reversed. Capability trade-offs covers two findings about achieving useful persona control while preserving general capabilities, and how stronger interventions can impair them. Robustness covers three findings about reproducing control across base models and data-generating teachers, and whether the results still hold with a simpler weight adjustment. In addition to the existing baselines, we also evaluated the paper generated in [1] using OpenClaw (Claude Opus 4.8) on the same task, applying our evaluation system for comparison.

The results are as follows:

MethodPersona Control (%)Capability Trade-off (%)Robustness (%)Overall Score (%)
Station (Seed 1)71.1172.2237.0461.11
Station (Seed 2)80.0066.6733.3363.33
Station (Seed 3)71.1138.8933.3353.33
Station (Overall Mean)74.0759.2634.5759.26
OpenClaw (Claude)40.000.0033.3330.00
Codex Multiagent-v2 (GPT)40.0050.0033.3340.00
AI Scientist-v2 (GPT)20.0050.000.0020.00
AI Scientist-v2 (Gemini)40.0066.670.0033.33
AI Scientist-v2 (Claude)80.000.0033.3350.00

The research reports and dialogue logs from the Station runs have been uploaded to the repository for transparency.

We observe that Station recovers more findings than the baselines, with contributions across all three categories. Our review of Station’s results shows examples of building on initial observations through further investigation: agents initially found that combining different trait adjustments in weight space did not produce the intended mixture of traits in the model’s behavior. Based on this finding, they separated out changes shared by the adjustments and tuned how the adjustments were combined to reduce interference between traits, making their combined effects more predictable. In contrast, Kirgis et al. [1] report that their OpenClaw system committed prematurely to one approach and struggled to turn initial observations into meaningful downstream findings. We observed similar difficulties in the baselines. As such, this task provides further evidence that a suitable research environment can help agents build on intermediate findings to make deeper discoveries in open-ended research. We will add this experiment and its details to the revised version of the paper.

Finally, we would also like to point out that open-ended autonomous discovery remains considerably expensive. With the additional experiment above, we now have four rediscovery tasks, each with three seeds and one Station run plus four baseline runs, for a total of 4 × 5 × 3 = 60 runs, in addition to the two exploratory tasks. The existing evaluation had already consumed over 760 million output tokens and 3,400 hours of cumulative experiment time, as reported in Appendix E.3. In comparison, similar work, such as [1], reports an evaluation based on only two tasks. Although we agree that adding more tasks would strengthen the evaluation, doing so is currently beyond our computational budget, and we hope the reviewer understands.

Comments on the paper

4 comments

Dear Reviewers,

As the discussion phase draws to a close, please acknowledge the author's response and let them know whether it has addressed some or all of your concerns.

Thank you for the time and effort you and the other reviewers have put into this discussion.

Best, Your AC

Dear AC,

Thank you for reminding the reviewers.

Dear reviewers,

We have devoted substantial effort to addressing your concerns, including conducting new experiments and providing detailed responses. With one day remaining in the discussion phase, we would greatly appreciate your response. Your feedback would also give us an opportunity to respond to any remaining questions before the discussion closes.

Best regards,

The authors

We thank all reviewers for reviewing the manuscript and the AC for sending the reminder. As the discussion period is coming to an end, we summarize the reviewers’ views and our rebuttal below for the convenience of the AC and the public:

We are pleased that the reviewers recognize the following strengths of our work:

1. Addressing Important Questions: Reviewers (wCWS, RoFx) point out that the question of open-ended scientific discovery addressed by our paper is important and timely. They recognized this to be a “crucial and highly difficult problem”.

2. Strong Empirical Performance: Reviewers point out that the proposed system has strong empirical performance, such as beating baselines by a significant margin (nxPh), and note that “the empirical results on the exploratory tasks are particularly impressive, as the system’s ability to independently discover actionable insights that match concurrent human work highlights its practical scientific utility” (wCWS).

3. Rigorous Evaluation Methodology and Extensive Empirical Analysis: Reviewers point out that the paper proposed a strong evaluation methodology for open-ended scientific discovery, e.g., a “rigorous and transparent evaluation methodology” (wCWS) or a “concrete protocol” (nxPh), accompanied by extensive empirical analysis (RoFx).

Besides these, reviewers also point out that the proposed method is “highly creative and innovative” (wCWS), and the paper is well-written (the writing is clear, according to RoFx, and the authors “exhibit commendable candor”, according to wCWS).


We summarize the reviewers’ concerns and our responses below:

1. Insufficient task evaluation: Reviewer nxPh and Reviewer RoFx commented that the tasks we report on are not extensive enough.

We would like to point out that the paper already covers 5 tasks, costing over 760 million output tokens and involving over 50 research runs due to various baselines and multiple seeds, and already covers more tasks than a typical empirical paper that studies open-ended scientific discovery (e.g., [1] covers 2 tasks only). Though it is always helpful to add more evaluation tasks and more seeds, authors have limited resources that need to be taken into account. Nonetheless, we still added one task from [1], making our paper cover 6 tasks in total. The details of the task results can be found in Part 3 of our response to Reviewer nxPh.

For reference, we listed recent AI scientist papers that stated their total output token use, showing that the total output token consumption of our paper is much higher than that of recent works:

PaperConferenceTotal output tokensSource
MLR-Bench [2]NeurIPS 2025 D&B5,162,423Table 7
PaperBench [3]ICML 2025≈40,000,000Sec. 4.1, footnote 7
Grow a Research World for AI-Scientists [4]arXiv 2026231,000,000Sec. 3.1, Table 1
MASS: Recursive Self-Improvement through Multi-Agent Self-Supervision [5]arXiv 2026≈29,787,240App. D.3
Can AI agents conduct open-ended AI research? Early evidence from two case studies [1]arXiv 2026≈17,000,000Secs. 3, 6
Ours--->760,000,000App. E.3, Table 2

2. Insufficient ablation on the proposed mechanisms: Reviewer RoFx pointed out that though we have ablation results for the two proposed mechanisms (Meta Reflection and Supervisor) together, we did not have ablation results for them individually.

We agree that this is a valid concern. As such, we added individual ablation experiments, which show the individual components’ effects on performance. Detailed results can be found in Part 2 of our response to RoFx.

3. Clarification on Misunderstandings: Besides, there are also several misunderstandings in the reviews involving factual or logical errors about our work or prior work.

a. Our token accounting may be wrong: One reviewer (RoFx) pointed out that the baselines (AI-Scientist v2 and OpenAI multi-agent v2) are ‘single-workflow’, so their token usage should be much less than that of Station, which has multiple agents. We would like to clarify that both baselines (AI-Scientist v2 and OpenAI multi-agent v2) use multiple agents in parallel in their workflows, similar to Station. We also confirm that the token usage we report includes all tokens used by the entire Station system and does not miss anything.

b. The system is very expensive: One reviewer (wCWS) pointed out that “the system requires a massive computational budget of thousands of GPU hours and millions of dollars in API costs to rediscover a single paper’s findings”. This limits how it can be applied to daily researcher use. We would like to clarify that a single Station run on a single problem only costs around USD 500 in API costs and at most USD 100 in GPU rental costs.

c. The original Station is already strong compared to the baselines: One reviewer (nxPh) points out that the original Station is already strong compared to the baselines and believes this to be a weakness. We would like to clarify that this paper’s goals are (i) empirically investigating existing AI systems in open-ended discovery and (ii) adapting them by proposing further improvements. The original Station was designed and evaluated for scorable tasks with explicit evaluation scores, and its performance on open-ended tasks was not known prior to this paper. There exists a similar concurrent paper that investigates existing AI systems such as OpenClaw on open-ended scientific discovery [1]. As such, logically, showing that the original Station works well in open-ended discovery should not be considered a weakness.

The reviewers also question why Station appears much stronger than the baselines, which we followed up on by running an additional task and performing additional analysis. Our analysis is consistent with concurrent findings from [1] and suggests that centralized systems may limit the diversity of exploration. See Part 1 of our response to Reviewer nxPh for details. The reviewers also requested other experiments (e.g., applying the same consolidating agent to the baselines and using the same model in Station), which we promptly conducted.


We believe that all issues raised by the reviewers are thoroughly addressed in our rebuttal, while the important strengths recognized by the reviewers remain. As such, we hope the AC takes account of these factors in the final judgement of our paper.

[1] Kirgis, P., et al. (2026). Can AI agents conduct open-ended AI research? Early evidence from two case studies. arXiv preprint arXiv:2607.27191. https://arxiv.org/abs/2607.27191

[2] Chen, H., et al. (2025). MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research. NeurIPS 2025, Datasets and Benchmarks Track. https://arxiv.org/abs/2505.19955

[3] Starace, G., et al. (2025). PaperBench: Evaluating AI’s Ability to Replicate AI Research. ICML 2025. https://arxiv.org/abs/2504.01848

[4] Lu, T., Konan, S., and Liu, Z. (2026). Grow a Research World for AI-Scientists. arXiv preprint arXiv:2611.08478. https://arxiv.org/abs/2611.08478

[5] Lee, H., et al. (2026). Recursive Self-Improvement through Multi-Agent Self-Supervision. arXiv preprint arXiv:2610.12176. https://arxiv.org/abs/2610.12176

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.