acceptodds
Under review as a conference paper at ICLR 2027

ScientistTwo: Pioneering the Human Knowledge Frontier with Autonomous AI

Abstract

Scientific discovery is defined by the ability to identify the boundaries of existing knowledge and venture into unexplored territory. The ultimate vision for AI in science is problem-driven autonomous research: given a challenge by a human expert, the AI independently navigates the scientific landscape, uncovers bottlenecks, and systematically expands the frontier of knowledge. In this paper, we introduce ScientistTwo, a fully autonomous multi-agent framework designed to realize this vision. Specifically, ScientistTwo takes an initial problem as input, establishes state-of-the-art baselines, formulates novel hypotheses, and coordinates specialized agents to orchestrate an end-to-end discovery cycle without human intervention. Moreover, the framework rigorously conducts experiments using diverse datasets and metrics, refines methodologies through automated ablation studies, and validates research findings via a closed-loop simulated peer-review rebuttal engine. To evaluate ScientistTwo's capabilities against the highest standards of human scientific achievement, we benchmark it across papers accepted at top-tier venues such as ICLR, ICML, and NeurIPS. As a result, ScientistTwo autonomously generates publishable papers and fully verified codebases. Its solutions consistently outperform the state-of-the-art models, and surpasses human-authored papers in evaluations by both AI review agents and human reviewers. These results show that ScientistTwo is not merely an assistive tool but an autonomous scientific pioneer capable of pushing the human knowledge frontier.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Autonomous Research Agents, AI Agents, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_2dq5OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper is a comprehensive study that, through large-scale experiments, demonstrates the extent to which current state-of-the-art AI Scientist systems can improve upon existing papers from top conferences. Specifically, they built a system called “ScientistTwo”, which conducts automated research through novel, detailed experimental designs (such as subset experiments and full experiments) as well as mechanisms like the Rebuttal Agent. As a result, they succeeded in improving the performance of existing papers in 86 out of 107 papers.

Critical strengths
  • S1. I think it is impressive that the experiment was conducted on such a large scale. While existing papers generally extend their methods to cover 10 or fewer papers, this paper extends it to 100 papers and demonstrates higher effectiveness.

  • S2. I found the addition of the Rebuttal Agent to be particularly innovative. While there have been methods in the past that improved papers based on reviews, none had incorporated this into the experimental process itself.

  • S3. The paper includes detailed experimental evaluations. It is impressive not only that the review score has improved compared to existing AI-generated papers, but also that the effectiveness of the Rebuttal Agent has been verified and an integrity audit has been conducted.

Critical weaknesses

I highly evaluate the abundant experimental results, research findings, and the design details of the framework presented in this paper. However, my concerns include the following:

  • W1. Unclear Differences from and Novelty Relative to Existing Research

This paper, primarily in the Abstract and Introduction, describes systematic ablation studies, multi-metric evaluations, and improvements over existing top-conference research as if they were new contributions. However, these have already been achieved in existing research such as AI Scientist-v2 and Jr. AI Scientist [1] for ablation, and DeepScientist and Jr. AI Scientist [1] for improvements over existing top-conference papers. It is necessary to clearly articulate the unique technical novelty of ScientistTwo without underestimating the contributions of prior research.

  • W2. Concerns Regarding Exaggeration of Scientific Claims and Hallucinations

Although the Integrity Audit verifies the consistency between experimental results and the code, the extent to which exaggeration of novelty, overgeneralization of experimental results, and hallucinations occur has not been sufficiently analyzed. In particular, further investigation is needed into whether repeated paper revisions by the Rebuttal Agent might exacerbate these issues.

  • W3. Insufficient Evaluation of the Scientific Validity of the Research Methodology Itself

While this paper primarily evaluates performance improvements, it has not sufficiently verified whether the generated research methodology is scientifically valid or truly valuable as research. It is important not only to improve scores but also to evaluate the scientific reliability of the proposed methodology itself.

[1] Miyai+, Jr. AI Scientist and Its Risk Report: Autonomous Scientific Exploration from a Baseline Paper, TMLR2026

Questions for authors

Refer to Critical Weaknesses.

5 comments

Deer Reviewer 2dq5

We deeply appreciate your invaluable time and efforts in reviewing our manuscript. We respond to each of your comments one-by-one in what follows. The revised manuscript has been marked in red.


[W1] Unclear Differences from and Novelty Relative to Existing Research

We thank the reviewer for the feedback. We agree that our Abstract and Introduction should more precisely highlight our specific technical novelties. We will thoroughly revise the Introduction and Related Work to properly cite the prior frameworks while clearly isolating ScientistTwo’s novel system architecture.

Specifically, first, while DeepScientist [1] formulates discovery as Bayesian Optimization over a single target objective, ScientistTwo evaluates and selects ideas jointly across multi-dataset, multi-metric benchmarks. Second, complementing DeepScientist’s surrogate-guided progressive filtering, ScientistTwo pairs subset-first screening with an engineering loop–tuning promising ideas on representative slices before scaling compute to full benchmarks, preventing brute-force resource waste and significantly improving the efficiency.

Third, we extend both how ablations guide discovery and how peer-review critiques are resolved. While AI Scientist-v2 [2], Jr. AI Scientist [3], and DeepScientist execute systematic ablations, they primarily run them after selecting a candidate method to analyze and tabulate component effects for paper writing. ScientistTwo turns ablation into a closed-loop refinement step: an Ablation Critic checks component-level changes to prune unhelpful modules, and re-engineer the method before drafting (see Table 8 of the main manuscript for the qualitative result). Finally, beyond automated review scoring or text-only reflection, our Rebuttal Agent writes and runs targeted supplementary experiments before paper finalization. Notably, while Jr. AI Scientist explicitly notes that expanding comparative methods (selecting and accurately reproducing baselines) remains beyond current autonomous AI Scientists and still requires human intervention, our Rebuttal Agent autonomously introduces and runs new baselines and held-out datasets without human input (e.g., see pages 69–73 for the example). Overseen by a Meta-Review Agent alongside automated integrity audits, this also directly mitigates the specification-gaming and citation risks highlighted in Jr. AI Scientist.

Together, these mechanisms represent a transition from code-and-write pipelines to an adaptive, self-auditing research agent. We explicitly clarified these distinctions and cited Jr. AI Scientist additionally, in the revised manuscript (see Section 2).

[1] Weng et al., DeepScientist: Advancing Frontier-Pushing Scientific Findings Progressively, 2026.

[2] Yamada et al., The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search, 2025.

[3] Miyai et al., Jr. AI Scientist and Its Risk Report: Autonomous Scientific Exploration from a Baseline Paper, 2026.


[W2] Concerns Regarding Exaggeration of Scientific Claims and Hallucinations

We thank the reviewer for this insightful comment regarding the critical need to guard against exaggerated claims and overgeneralization during autonomous paper generation and rebuttal.

First of all, we would like to clarify that the Rebuttal Agent's workflow is naturally designed to prevent exaggeration during paper revisions; its primary function is conditioned on execution feedback, where it autonomously designs, executes, and integrates targeted supplementary experiments to address critique. Because any textual revisions must directly reflect new empirical outputs verified by our execution logs, the iterative rebuttal cycle does not amplify ungrounded claims or hallucinated gains. Moreover, as shown in Table 5 of our manuscript, it is worth noting that the paper generated by ScientistTwo faithfully and accurately describes the codebase implementation and confirms that the methodology section is well explained without exaggeration. In addition, ScientistTwo is also guaranteed that the bibliography contains no hallucinated citations.

Second, to further alleviate your concern, we conducted a targeted audit evaluating: (1) novelty claim, and (2) whether experimental results are overgeneralized. Across all 33 papers evaluated on problem setups drawn from NeurIPS 2025, only two papers were flagged for mildly overgeneralized framing or overstated novelty. This confirms an empirically low rate of exaggeration (~6%) under the current pipeline.

We fully agree that systematically auditing text-level semantic integrity alongside code-level numerical consistency is essential for trustworthy scientific agents. We included these discussions and experiments in the revised manuscript (see Appendix B.3).

[W3] Insufficient Evaluation of the Scientific Validity of the Research Methodology Itself

We appreciate the reviewer’s thoughtful feedback regarding the distinction between empirical benchmark gains and fundamental scientific validity. We fully agree that higher performance metrics alone do not guarantee sound research methodology, which is precisely why our evaluation framework explicitly decoupled empirical outcomes from methodological rigor.

Rather than relying solely on performance gains, we evaluated scientific validity of ScientistTwo’s proposed methods through dedicated, blinded human expert evaluations. As presented in Table 3 of our main manuscript, human experts evaluated the generated research across six core scientific dimensions, explicitly rating Methodological Soundness (averaging 4.3 / 5.0) and Scientific Maturity (averaging 4.1 / 5.0). Crucially, human evaluators examined whether the underlying hypotheses were theoretically motivated, whether the experimental designs adhered to rigorous scientific control standards, and whether the algorithmic contributions avoided trivial shortcuts or conceptual flaws.

To complement human evaluation at scale, we evaluated our generated papers against published baselines using both ScholarPeer and the Stanford Agentic Reviewer–AI reviewers that audits technical soundness, experimental validity, academic values, etc. Moreover, we would like to note that both ScholarPeer and the Stanford Agentic Reviewer demonstrate significant alignment with human review: ScholarPeer achieves a state-of-the-art Spearman rank correlation of 0.43 with human reviewers [1], while the Stanford Agentic Reviewer achieves 0.42 with human scores–exceeding 0.41, a typical human-to-human inter-annotator agreement [2] and predicts paper acceptance with an AUC of 0.75. This empirical baseline supports the premise that these simulated review scores serve as meaningful, non-random proxies for scientific quality. Using these AI reviewers, as shown in Table 1 of our manuscript, ScientistTwo-generated papers exceeded the average scores of accepted human-written manuscripts from ICLR 2026 and NeurIPS 2025.

[1] Goyal et al., ScholarPeer: A Context-Aware Multi-Agent Framework for Automated Peer Review, 2026

[2] Jiang and Ng, Tech Overview, papperreview.ai/tech-overview

AuthorsOpenReview ↗

Further Discussion Before the Deadline

Dear Reviewer 2dq5,

Thank you for your time and efforts again in reviewing our paper.

We kindly remind that the discussion period will end soon.

We believe that we sincerely and successfully address your comments, with the results of the supporting experiments.

If you have any further concerns or questions, please do not hesitate to let us know.

Thank you very much!

Authors

Reviewer_pXCiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ScientistTwo, a multi-agent framework for autonomous scientific research. The topic is important, and the system demonstrates substantial engineering effort. However, I have serious concerns about the reproducibility, fairness of comparisons, and validity of the evaluation. The experimental evidence does not adequately support the paper's strong claims about surpassing existing AI Scientist systems or human scientific research.

Critical strengths

Important research problem. Developing autonomous agents capable of conducting rigorous scientific research is a timely and meaningful direction.

Broad experimental coverage. Evaluating 107 research tasks across ICLR, ICML, and NeurIPS papers is ambitious. The inclusion of human evaluation and code-integrity audits is also appreciated.

Critical weaknesses

Unclear and unverifiable SOTA improvement claims. Figure 1 prominently reports an 80.4% success rate and a 25.2% average relative improvement over human SOTA. However, the paper does not clearly explain how relative improvements are calculated and aggregated across tasks with different evaluation metrics. The normalization and upper-bound definitions in Figure 1(a) are also unclear. Without per-task results and explicit calculation rules, these central claims cannot be independently verified.

Unfair comparisons with existing AI Scientist systems. Table 1 compares ScientistTwo against papers generated by previous systems under different models, computational budgets, and experimental conditions. Notably, ScientistTwo uses Claude Opus 4.8, a substantially more recent and capable model than those available to many baseline systems, along with a sophisticated multi-agent pipeline and approximately $3,765 per task. These differences make it impossible to determine whether the reported improvements come from the proposed framework or simply stronger foundation models and greater computational resources. Controlled comparisons using the same models, tasks, and budgets are essential to establish the actual contribution of ScientistTwo.

AI reviewer scores do not establish scientific quality. The paper relies heavily on automated review scores and predicted acceptance rates to support claims of human-level or superior scientific research. However, these scores cannot reliably establish scientific novelty, methodological soundness, or the validity of research findings. Importantly, the human evaluation in Table 3 reports overall parity with human-authored papers, not superiority. The conclusions substantially overstate what the evidence demonstrates.

Questions for authors

Can the authors release the complete prompts, agent configurations, and evaluation criteria necessary to reproduce ScientistTwo?

How exactly are the relative SOTA improvements in Figure 1 calculated across heterogeneous tasks? Can the authors provide the complete per-task results underlying the reported 25.2% average gain?

Can the authors compare ScientistTwo with representative baseline systems using the same foundation models, tasks, and computational budgets? In particular, how does ScientistTwo compare with a general-purpose coding agent given an equivalent $3,765 budget?

What evidence demonstrates that the generated research constitutes genuine scientific contributions beyond benchmark improvements and favorable AI reviewer scores? Can the authors provide independent expert assessments of methodological novelty and scientific validity?

4 comments

Deer Reviewer pXCi

We deeply appreciate your invaluable time and efforts in reviewing our manuscript. We respond to each of your comments one-by-one in what follows. The revised manuscript has been marked in red.


[W1, Q2] Clarification of relative gain calculations and per-task results.

Thanks for the comment. To measure the relative gain, we first identify the baseline human SOTA or previous top baseline score (base) and our method’s score (ours) for each task. The task-level relative improvement is computed deterministically as (ours - base) / |base| x 100 for higher-is-better metrics, and (base - ours) / |base| x 100 for lower-is-better metrics. Here we use Gemini 3.6 Flash to parse and extract the metric from the experimental logs. To account the reasoning variance during generation, we run the evaluation pipeline 10 times per task and report the average performance. Below is the example:

1. Selected metric: Average Test Mean Absolute Error (MAE) across all 4 benchmark regression datasets (dsprites_simple, pendulum, mnist_arithmetic, mawps) from the reproduced experiments.
2. Collected Test MAE scores for each model across datasets:
   - sym_m_cbe (original method): [dsprites_simple: 0.998, pendulum: 0.911, mnist_arithmetic: 3.411, mawps: 1.507] -> Mean = 1.70675
   - flex_dot_m_cbe (our method): [dsprites_simple: 0.917, pendulum: 0.795, mnist_arithmetic: 2.304, mawps: 1.001] -> Mean = 1.25425
3. Calculated relative gains (% improvement in MAE reduction):
   - relative_gain_our_method = |1.70675 - 1.25425| / |1.70675| * 100 = 26.5124%

The reported 25.2% overall gain in Figure 1 of our main manuscript represents the average of these relative improvements across all tasks.

For normalization, we extracted the required relative gain to achieve the upper bound of the benchmark through the same procedure. Then, we calculated human state-of-the-art performances as 1 / (1 + required relative gain), and ScientistTwo’s performance by (1 + our relative gain) / (1 + required relative gain).

In the revised manuscript, we added the full per-task breakdown table covering all tasks (see Appendix G of our revised manuscript), and the evaluation scripts to the Appendix J.11 so that all claims can be independently reproduced. In addition, the generated papers were also provided in the supplementary materials.

[W2, Q3] Controlled baseline comparisons.

Thank you for raising this critical point. To isolate the contribution of our multi-agent framework from foundation model strength and compute budgets, we conducted two controlled experiments during the rebuttal period.

First, we re-evaluated ScientistOne (our strongest baseline in Table 1 of the manuscript) using the same foundation models as ScientistTwo (Gemini 3.6 Flash and Claude Opus 4.8). Because ScientistOne is structurally limited to single-metric optimization–whereas ScientistTwo validates ideas across multiple benchmarks and metrics–we evaluated ScientistOne and ScientistTwo on five representative tasks from the ScientistOne’s benchmark. Here, we let 20 trials for idea validation for each framework, following the experimental setup of the ScientistOne paper. As shown below, upgrading ScientistOne to Opus 4.8 improved its raw scores, yet it still lagged significantly behind ScientistTwo in Stanford Agentic Reviewer (SAR) scores and acceptance rates, demonstrating that stronger models alone do not bridge the gap.

\begin{array}{lccc} \hline \textbf{Framework} & \textbf{Model} & \textbf{Avg. Rating} & \textbf{Accept Rate} \newline \hline \text{ScientistOne} & \text{Gemini 3.6 Flash} & 4.1 & 0\% \newline \text{ScientistOne} & \text{Claude Opus 4.8} & 4.6 & 0\% \newline \textbf{ScientistTwo (Ours)} & \text{Gemini 3.6 Flash \& Claude Opus 4.8} & \textbf{5.9} & \textbf{60\%} \newline \hline \end{array}

Second, we compared ScientistTwo against Claude Code (powered by Claude Opus 4.8) on 10 tasks drawn from NeurIPS 2025 (selected for runtimes under 100,000 seconds due to rebuttal constraints). We provided Claude Code with the identical baseline code, paper, and per-task API budget used by ScientistTwo, prompting it to analyze the work and autonomously develop a superior method. While Claude Code proved to be a strong baseline, ScientistTwo outperformed it across every metric, including average review score, acceptance rate, and relative gain. These controlled comparisons confirm that ScientistTwo’s gains stem primarily from its structured multi-agent design rather than model choice or budget alone. Moreover, qualitatively, as shown in Figure11 of Appendix F.2 of the revised manuscript, our framework demonstrates substantially broader experimental coverage and analytical rigor. While Claude Code generated only one table, two figures, and four references, ScientistTwo produced 12 tables, 6 figures, and 51 references.

\begin{array}{lcccccc} \hline \textbf{Framework} & \textbf{Model} & \textbf{Avg. Rating (ScholarPeer)} & \textbf{Accept Rate (ScholarPeer)} & \textbf{Avg. Rating (SAR)} & \textbf{Accept Rate (SAR)} & \textbf{Avg. Gain} \newline \hline \text{Claude Code} & \text{Claude Opus 4.8} & 5.0 & \text{50\%} & \text{5.57} & \text{60\%} & \text{5.97\%} \newline \textbf{ScientistTwo (Ours)} & \text{Gemini 3.6 Flash \& Claude Opus 4.8} & \textbf{8.0} & \textbf{100\%} & \textbf{5.72} & \textbf{90\%} & \textbf{19.69\%} \newline \hline \end{array}

We included these additional experiments in Appendix F of our revised manuscript.

[W3, Q4] Response to concerns regarding scientific validity and validation beyond AI scoring.

First of all, we would like to note that both ScholarPeer and the Stanford Agentic Reviewer demonstrate significant alignment with human review: ScholarPeer achieves a state-of-the-art Spearman rank correlation of 0.43 with human reviewers [1], while the Stanford Agentic Reviewer achieves 0.42 with human scores–exceeding 0.41, a typical human-to-human inter-annotator agreement [2] and predicts paper acceptance with an AUC of 0.75. This empirical baseline supports the premise that these simulated review scores serve as meaningful, non-random proxies for scientific quality.

Secondly, regarding human evaluation in Table 3 of our main manuscript, achieving parity is a strong result rather than a limitation. Specifically, we asked “Considering the balance between claims, empirical evidence, and clarity, which paper demonstrates higher technical maturity and readiness for acceptance at a venue like ICLR?” to measure overall scientific maturity and top-tier readiness. Here, our generated manuscripts were evaluated against published, peer-reviewed NeurIPS papers (i.e., accepted ones; not just submitted) whose scientific rigor had already passed human acceptance thresholds. Achieving comparable quality–while outperforming these published baselines in experimental design, benchmarks, and ablations–indicates that our framework produces substantive scientific contributions. In Appendix I of the revised draft, we included the questions used for human evaluation.

Finally, to address concerns about genuine methodological soundness beyond automated metrics, we would like to note that we have already provided domain-expert case study in Appendix C, which evaluates the concrete scientific artifacts produced by ScientistTwo. Specifically, a domain expert with extensive background in time-series conducted a rigorous peer-review style evaluation of the generated paper, critically assessing the novelty, correctness, and empirical validity of the proposed method. Rather than relying on trial-and-error heuristics, the agent autonomously identified failure modes in retrieval-augmented time-series forecasting, formulating a mathematically grounded, frequency-aware gating mechanism (DynaSpec-RAG) that achieved zero-shot transfer with minimal parameter overhead. While logistically constrained by time and domain-expert recruitment during the current rebuttal phase, we are currently working to scale up the expert evaluation and will incorporate these expanded qualitative results into the final draft.

[1] Goyal et al., ScholarPeer: A Context-Aware Multi-Agent Framework for Automated Peer Review, 2026

[2] Jiang and Ng, Tech Overview, papperreview.ai/tech-overview


[Q1] Can the authors release the complete prompts, agent configurations, and evaluation criteria necessary to reproduce ScientistTwo?

Yes, we incorporated complete prompt templates in the Appendix J of the revised manuscript, including prompts for evaluation. See Appendix G.2 of our main manuscript for the agent configurations. Additionally, we plan to release the evaluation criteria for each task as open source.

AuthorsOpenReview ↗

Further Discussion Before the Deadline

Dear Reviewer pXCi,

Thank you for your time and efforts again in reviewing our paper.

We kindly remind that the discussion period will end soon.

We believe that we sincerely and successfully address your comments, with the results of the supporting experiments.

If you have any further concerns or questions, please do not hesitate to let us know.

Thank you very much!

Authors

Reviewer_crs3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ScientistTwo, an end-to-end autonomous multi-agent research framework that identifies limitations in existing methods for a given research problem, generates and evolves hypotheses, conducts experiments from benchmark subsets to full benchmark suites, performs ablation studies, and drafts research papers. The framework further refines its research outputs through simulated peer-review, rebuttal, and meta-review loops. Evaluated on 107 research tasks derived from accepted papers at ICLR, ICML, and NeurIPS, ScientistTwo demonstrates strong performance across diverse research domains, achieving substantial improvements over human state-of-the-art baselines, high acceptance rates from automated review agents, and strong consistency between generated papers and codebases, as demonstrated by integrity audits.

Critical strengths
  1. ScientistTwo simulates multiple stages of the scientific research process, including idea generation, experimental evaluation, hypothesis refinement, multi-agent collaboration, and peer review. This provides a more comprehensive approach to autonomous scientific research compared to existing methods that primarily focus on optimizing a single performance metric.
  2. The framework emphasizes evaluation across multiple datasets and metrics rather than relying on a single benchmark score. It also incorporates multiple validation mechanisms, including ablation studies, simulated peer review, and integrity audits, to improve the rigor and reliability of the generated research.
  3. This paper evaluates ScientistTwo on 107 research tasks derived from papers accepted at top-tier conferences, covering diverse research domains. The evaluation also includes comparisons with existing autonomous research systems and human-authored papers, supported by both automated and human reviewers, providing relatively comprehensive evidence of the framework's effectiveness.
Critical weaknesses
  1. In Section 3.1, the framework relies on a Limitation Extractor and a Limitation Verifier to identify weaknesses in existing human SOTA methods. However, it is unclear whether the Limitation Verifier assesses the factual validity of these limitations or merely their sufficiency for generating new ideas. Without explicit empirical or theoretical evidence supporting the identified limitations, the framework may generate hypotheses based on plausible but unverified assumptions.

  2. The authors criticize prior autonomous research agents for focusing on single-metric optimization and emphasize the importance of holistic, multi-metric evaluation. However, improving performance across multiple benchmarks does not necessarily imply meaningful scientific discovery. In ScientistTwo, idea selection and refinement remain largely driven by empirical performance feedback. Although the framework incorporates novelty checking and ablation studies, it is unclear how these mechanisms establish the scientific significance of a proposed idea beyond performance improvements.

  3. The framework heavily relies on simulated peer-review and meta-review agents to refine manuscripts and assess their scientific quality. Although it includes a held-out AI reviewer and human evaluations, it remains unclear how reliably simulated review scores reflect genuine scientific quality. Notably, Table 4 shows that increasing review rounds improves the acceptance rate under ScholarPeer but does not consistently improve performance under the held-out Stanford Agentic Reviewer. Moreover, the human evaluation in Table 3 indicates overall parity with human-authored papers rather than clear superiority.

Questions for authors
  1. How does the Limitation Verifier assess the validity of identified limitations? Does it verify whether the identified limitations actually exist based on empirical or theoretical evidence, or does it primarily evaluate whether they are sufficient to guide idea generation? Could the authors provide more details about the verification criteria and a concrete example?

  2. How does ScientistTwo distinguish meaningful scientific contributions from empirical performance improvements? Although the framework incorporates novelty checking and ablation studies, idea selection and refinement still appear to be largely driven by benchmark performance. Could the authors clarify how the scientific significance of generated ideas is evaluated beyond improvements in performance metrics?

  3. How reliably do simulated peer-review scores reflect actual scientific quality? In Table 4, additional review rounds improve the acceptance rate under ScholarPeer but do not consistently improve the results from the held-out Stanford Agentic Reviewer. Could the authors explain this discrepancy and discuss whether the framework may over-optimize toward the simulated reviewer?

  4. Could the authors provide ablation studies on the Limitation Verifier and Novelty Checker? While the paper evaluates the effectiveness of the peer-review process and integrity refinement agents, the individual contributions of these two components in the idea generation stage remain unclear. It would be helpful to investigate how removing these components affects the novelty, quality, and performance of generated ideas under comparable computational budgets.

4 comments

Deer Reviewer crs3

We deeply appreciate your invaluable time and efforts in reviewing our manuscript. We respond to each of your comments one-by-one in what follows. The revised manuscript has been marked in red.


[W1, Q1, Q4] Clarification on the Verification Criteria of the Limitation Verifier & Ablation Study on the Limitation Verifier.

We thank the reviewer for this insightful comment. In our design, the two modules play complementary roles: while the Limitation Verifier evaluates whether an identified limitation provides sufficient actionable insight to guide meaningful idea generation, the factual grounding is naturally provided by the Limitation Extractor (see examples on pages 52–53 of our main manuscript). Below, we provide core criterias of the Limitation Verifier and an example of the extracted limitation.

Criteria.

  • Enough extract limitations, if the given limitations are enough to guide a novel improvement of the method.
  • Otherwise, find the limitations of the method which are not mentioned previously.
  • Limitation should focus on methodology, not a lack of experiments.

Example: Static Past-Only Distance Metric for Retrieval (Context-Future Divergence)

  • Limitation: The retriever identifies top-k nearest neighbors based solely on Euclidean distance in the latent space of the past context (xqx_q vs. xix_i).
  • Reason: In complex and non-stationary time series, similar historical contexts (xq≈xix_q \approx x_i) often lead to vastly different future trajectories (yq≠yiy_q \neq y_i) due to unobserved variables, regime shifts, or chaotic dynamics.

Moreover, to further address your concern, we conducted an additional ablation study on the Limitation Verifier. Due to the time constraints of the rebuttal phase, we selected 10 NeurIPS 2025 tasks that required under 100,000 seconds to run. As shown in the table below, removing the Limitation Verifier degraded both the acceptance rate and average review score when evaluated by the Stanford Agentic Reviewer. This confirms that sufficiently extracting limitations is key to guiding more novel research directions and achieving stronger performance. Nevertheless, even without the Limitation Verifier, our framework maintains reasonable performance because the Limitation Extractor already extracts well-grounded limitations of the human state-of-the-art methods.

\begin{array}{lcccc} \hline \textbf{Framework} & \textbf{Limitation Verifier} & \textbf{Avg. Rating} & \textbf{Accept Rate} & \textbf{Avg. Gain} \newline \hline \text{ScientistTwo (Variant)} & \text{X} & 5.57 & 80\% & 13.59\% \newline \textbf{ScientistTwo (Ours)} & \text{O} & \textbf{5.72} & \textbf{90\%} & \textbf{19.69\%} \newline \hline \end{array}

To make this verification criteria fully transparent, we provided the complete prompt templates (see Appendix J.2) and additional ablation studies regarding the Limitation Verifier (see Table 10 of Appendix B.2).

[W2, Q2, Q4] Evaluating scientific significance beyond empirical benchmark performance & Ablation Study on Novelty Checker.

We appreciate the reviewer’s thoughtful feedback regarding the distinction between empirical benchmark gains and fundamental scientific validity. We fully agree that higher performance metrics alone do not guarantee sound research methodology, which is precisely why our evaluation framework explicitly decoupled empirical outcomes from methodological rigor.

In order to make scientific contributions instead of just improving the performance, our ScientistTwo enforces novelty through a dedicated Novelty Checker prior to code generation. Specifically, (1) for any candidate idea, the agent retrieves closely related publications using Google Search as a tool. It explicitly parses and extracts their method sections to construct a direct contrastive context. (2) Also, using both internal parametric knowledge and retrieved literature, our Novelty Checker evaluates whether the candidate idea exhibits high overlap with known methods. (3) In addition, as an additional safeguard, the agent is guided to use Google Search as a tool to ensure no equivalent ideas exist.

To further alleviate your concern, we conducted an additional ablation study on the Novelty Checker. Due to the limited time of the rebuttal phase, we selected tasks that required less than 100,000 seconds, from NeurIPS 2025 problems (total 10 tasks). Specifically, in our framework, candidate ideas are scored by this Novelty Checker, and only those with relatively high novelty are implemented in descending order of their scores. As shown in the table below, removing the Novelty Checker degraded both the acceptance rate and average review score when evaluated by the Stanford Agentic Reviewer, since it increases the possibility to start with an idea that is less novel. This confirms that the Novelty Checker effectively filters out minor variations and prioritizes genuinely novel ideas with substantial scientific value.

\begin{array}{lcccc} \hline \textbf{Framework} & \textbf{Novelty Checker} & \textbf{Avg. Rating} & \textbf{Accept Rate} & \textbf{Avg. Gain} \newline \hline \text{ScientistTwo (Variant)} & \text{X} & 5.53 & 70\% & 13.68\% \newline \textbf{ScientistTwo (Ours)} & \text{O} & \textbf{5.72} & \textbf{90\%} & \textbf{19.69\%} \newline \hline \end{array}

Rather than relying solely on performance gains, we evaluated scientific validity of ScientistTwo’s proposed methods through dedicated, blinded human expert evaluations. As presented in Table 3 of our main manuscript, human experts evaluated the generated research across six core scientific dimensions, explicitly rating Methodological Soundness (averaging 4.3 / 5.0) and Scientific Maturity (averaging 4.1 / 5.0). Crucially, human evaluators examined whether the underlying hypotheses were theoretically motivated, whether the experimental designs adhered to rigorous scientific control standards, and whether the algorithmic contributions avoided trivial shortcuts or conceptual flaws. To complement human evaluation at scale, we evaluated our generated papers against published baselines using both ScholarPeer and the Stanford Agentic Reviewer (see Table 1 of our main manuscript), where ScientistTwo-generated papers systematically exceeded the average scores of accepted human-written manuscripts from ICLR 2026 and NeurIPS 2025.

In the revised paper, we included discussion and the result (see Table 10 and Appendix B.2), and provided the exact prompts (see Appendix J.4) for further understanding.

[W3, Q3] Reliability of Simulated Review Scores.

First of all, both ScholarPeer and the Stanford Agentic Reviewer demonstrate significant alignment with human review: ScholarPeer achieves a state-of-the-art Spearman rank correlation of 0.43 with human reviewers [1], while the Stanford Agentic Reviewer achieves 0.42 with human scores–matching or exceeding typical human-to-human inter-annotator agreement [2] and predicts paper acceptance with an AUC of 0.75. This empirical baseline supports the premise that these simulated review scores serve as meaningful, non-random proxies for scientific quality.

Second, because ScholarPeer guides the revision process (acting analogously to an optimization objective or training signal), evaluating with the held-out Stanford Agentic Reviewer provides an independent out-of-distribution test. Rather than failing to generalize, one round of revision using ScholarPeer dramatically improves the Stanford Agentic Reviewer’s acceptance rate from 49.0% to 73.5% (+24.5pp). The slight minor dip in the second round reflects typical diminished marginal returns and partial domain alignment rather than severe overfitting. Crucially, the substantial gain from round zero to round one demonstrates that acting on AI feedback yields genuine structural and technical improvements that transfer effectively across independently developed review models.

Furthermore, as shown in Table 4 of our main manuscript, ScientistTwo substantially outperforms baseline methods under both review agents even before any revision rounds are conducted.

Finally, regarding human evaluation in Table 3 of our main manuscript, achieving parity is a strong result rather than a limitation: our generated manuscripts were evaluated against published (accepted), peer-reviewed NeurIPS papers whose scientific rigor had already passed human acceptance thresholds. Achieving comparable quality–while outperforming these published baselines in experimental design, benchmarks, and ablations–indicates that our framework produces substantive scientific contributions.

[1] Goyal et al., ScholarPeer: A Context-Aware Multi-Agent Framework for Automated Peer Review, 2026

[2] Jiang and Ng, Tech Overview, papperreview.ai/tech-overview

AuthorsOpenReview ↗

Further Discussion Before the Deadline

Dear Reviewer crs3,

Thank you for your time and efforts again in reviewing our paper.

We kindly remind that the discussion period will end soon.

We believe that we sincerely and successfully address your comments, with the results of the supporting experiments.

If you have any further concerns or questions, please do not hesitate to let us know.

Thank you very much!

Authors

Reviewer_xgDFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ScientistT, a system designed to automate much of the scientific research process. It can identify limitations in existing work, generate research ideas, implement and test them, run ablations, write a manuscript, and revise it based on simulated peer review.

The overall direction is interesting. The main question is whether the system can produce genuinely useful and novel research, rather than simply assemble a polished research workflow.

Critical strengths

ScientistT does more than generate ideas or write papers. It tries to handle the whole process, from finding a problem to running experiments and revising the final manuscript.

The system uses failed experiments, ablation results, and reviewer feedback to decide what to try next.

If the reported results hold up, ScientistT could make research iteration much faster by automating many repetitive experiments.

Critical weaknesses

The reported scores are promising. Still, they may not capture correctness, insight, or the broader value of the research. A small amount of expert evaluation would make the results easier to trust.

The system appears to run many long experiments. This may be costly in practice. It would be useful to report the runtime, token usage, and total compute cost.

I would like to see more detail on how ScientistT avoids producing minor variations of well-known methods. It should also filter out ideas that sound reasonable but cannot be tested properly.

Some ideas may look strong on the initial subset but fail on the full benchmark. More evidence is needed to show that the subset-first strategy does not overfit to the screening set.

Questions for authors

How were the generated papers checked by human experts? It would be useful to know whether experts found them correct, novel, and scientifically meaningful.

What does a typical ScientistT run look like in terms of time and cost? Please report the approximate runtime, token usage, and compute resources.

The paper should clarify how ScientistT distinguishes genuinely new ideas from small variations of existing methods. Information about literature search or novelty checking would be useful here.

More details on the construction of the initial screening subsets would strengthen the evaluation.

3 comments

Deer Reviewer xgDF

We deeply appreciate your invaluable time and efforts in reviewing our manuscript. We respond to each of your comments one-by-one in what follows. The revised manuscript has been marked in red.


[W1, Q1] While scores are promising, expert human evaluation is needed to verify correctness, novelty, and scientific value.

We thank the reviewer for highlighting the importance of expert human evaluation. To directly assess correctness, novelty, and scientific impact, we have already conducted a systematic expert evaluation of 33 ScientistTwo-generated papers, presented in Table 3 of our main manuscript. Qualified experts with publication records at top-tier venues evaluated the papers across 12 targeted criteria. Specifically, five key questions directly addressed the reviewer’s core dimensions:

Novelty:

  • Q1: "Does the introduction clearly identify an authentic, non-trivial research gap and provide a mechanically sound motivation, rather than relying on circular or superficial problem statements?"

Correctness:

  • Q2: "Are the proposed method’s components logically justified with clear architectural or mathematical rationale, such that an ML researcher could understand the design choices without unexplained leaps?"

Broader Scientific Value & Rigor:

  • Q4: "Do ablation studies systematically isolate the impact of individual design components, demonstrating clear causal contribution rather than presenting trivial hyperparameter permutations?"
  • Q5: "Does the discussion extract actionable insights, analyze error modes, and acknowledge concrete boundary conditions, rather than merely verbalizing the numeric shifts in the tables?"
  • Q6: "Considering the balance between claims, empirical evidence, and clarity, does the paper demonstrate high technical maturity and readiness for acceptance at a venue like ICLR?"

On a 1-5 Likert scale (where 3.0 denotes neutral scores > 3.0 reflect positive evaluations), ScientistTwo-generated papers achieved:

  • Novelty (Q1): 4.2 / 5.0
  • Correctness (Q2): 4.1 / 5.0
  • Scientific Value & Rigor (Q4, Q5, Q6): 4.3 / 5.0, 4.0 / 5.0, and 3.7 / 5.0, respectively.

These scores indicate that human experts found the generated papers to be technically sound, well-motivated, and scientifically substantive. We included the questions used for human evaluation in Appendix I of our revised manuscript.


[W2, Q2] Concern regarding experimental runtime and computational cost (Runtime, Tokens, and Total Cost).

We thank the reviewer for highlighting the importance of reporting resource consumption. We agree that transparency regarding runtime, compute, and token costs is crucial for evaluating real-world feasibility.

First, an overview of the overall computational cost was included in Figure 4 of our main manuscript. Specifically, ScientistTwo requires an average of 2.5 days to complete the full research lifecycle, and costs an average of \3,765,includingtokenusagecosts(anaverageof3,765, including token usage costs (an average of \\1,666) and virtual machine costs (an average of \$2,099; mainly for 8 A100 40GB GPUs). To further alleviate the reviewer’s concern, we provide additional analysis regarding model token usage below.

\begin{array}{lcc} \hline \textbf{Model} & \textbf{Avg. Input Tokens} & \textbf{Avg. Output Tokens} \newline \hline \text{Gemini 3.6 Flash} & \text{5,691,314} & \text{478,853} \newline \text{Gemini 3 Flash Preview} & \text{332,901} & \text{45,925} \newline \text{Claude Opus 4.8} & \text{319,988,635} & \text{2,368,352} \newline \hline \end{array} We incorporated this table in Table 13 of Appendix E of our revised manuscript to ensure computational transparency.

[W3, Q3] Regarding Idea Novelty and Experimental Testability.

We thank the reviewer for raising these questions. First of all, ScientistTwo enforces novelty through a dedicated Novelty Checker prior to code generation. Specifically, (1) for any candidate idea, the agent retrieves closely related publications using Google Search as a tool. It explicitly parses and extracts their method sections to construct a direct contrastive context. (2) Also, using both internal parametric knowledge and retrieved literature, our Novelty Checker evaluates whether the candidate idea exhibits high overlap with known methods. (3) In addition, as an additional safeguard, the agent is guided to use Google Search as a tool to ensure no equivalent ideas exist. As a result, human experts rated the novelty of ScientistTwo’s generated ideas as 4.2 / 5.0 (detailed in our human evaluation questionnaire, [W1, Q1] above), aligning with our top-ranked AI review scores in Table 1 of our main manuscript.

To further alleviate your concern, we conducted an additional ablation study on the Novelty Checker. Due to the limited time of the rebuttal phase, we selected tasks that required less than 100,000 seconds, from NeurIPS 2025 problems (total 10 tasks). Specifically, in our framework, candidate ideas are scored by this Novelty Checker, and only those with relatively high novelty are implemented in descending order of their scores. As shown in the table below, removing the Novelty Checker degraded both the acceptance rate and average review score when evaluated by the Stanford Agentic Reviewer, since it increases the possibility to start with an idea that is less novel. This confirms that the Novelty Checker effectively filters out minor variations and prioritizes genuinely novel ideas with substantial scientific value.

\begin{array}{lcccc} \hline \textbf{Framework} & \textbf{Novelty Checker} & \textbf{Avg. Rating} & \textbf{Accept Rate} & \textbf{Avg. Gain} \newline \hline \text{ScientistTwo (Variant)} & \text{X} & 5.53 & 70\% & 13.68\% \newline \textbf{ScientistTwo (Ours)} & \text{O} & \textbf{5.72} & \textbf{90\%} & \textbf{19.69\%} \newline \hline \end{array}

Secondly, to eliminate ideas that sound plausible but cannot be tested empirically, ScientistTwo explicitly guides the Idea Generator to generate concrete execution specifications, including explicit algorithmic pseudocode (see pages 54–57 of our manuscript for an example). As a result, across all 107 evaluated tasks, 100% produced executable code that successfully completed empirical runs without human intervention. In addition, it is worth noting that our final idea and its implementation are well-aligned with each other, with no reward hacking and everything being reproducible (see Table 5 of our main manuscript).

In the revised paper, we included discussion and the result (see Table 10 and Appendix B.2), and provided the exact prompts (see Appendix J.4) for further understanding.


[W4, Q4] Concern regarding potential overfitting to the initial screening subset and requests for details on subset construction.

We appreciate the reviewer’s insightful comment. We would like to clarify that ScientistTwo does not rely on the initial subset to make final claims; rather, the subset-first strategy serves strictly as a cost-efficient preliminary filter, directly mirroring standard human research practices. When an idea shows promise on the screening set, it is immediately scaled up to the full benchmark for rigorous verification. Crucially, any idea that fails to generalize across the complete benchmark is pruned, ensuring that performance gains on the subset alone are never accepted as valid findings. It is worth noting that this subset-first strategy is possible because ScientistTwo utilizes multiple benchmarks, not just a single one, to ensure generalizability of the proposed idea.

Regarding the construction of the screening subset, ScientistTwo first reproduces the baseline method on a representative small subset of the benchmark and compiles an execution report specifying the exact data slice, splits, and seeds utilized. Using this standardized setup, ScientistTwo then evaluates the proposed idea under the identical subset configuration.

In our revised manuscript, we included the exact prompt templates in Appendix J.3 for further understanding.

AuthorsOpenReview ↗

Further Discussion Before the Deadline

Dear Reviewer xgDF,

Thank you for your time and efforts again in reviewing our paper.

We kindly remind that the discussion period will end soon.

We believe that we sincerely and successfully address your comments, with the results of the supporting experiments.

If you have any further concerns or questions, please do not hesitate to let us know.

Thank you very much!

Authors

Comments on the paper

1 comment
Area_Chair_BTbcOpenReview ↗

Please review the authors’ responses and join the discussion

Dear Reviewers,

The authors have responded to all reviews. Please read their responses and the other reviewers’ comments, then post a brief follow-up indicating which concerns, if any, remain and whether the responses have changed your assessment. If your concerns have been resolved, a short confirmation would also be helpful.

Thank you for your time and for contributing to the discussion.

Best regards,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.