- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper is a comprehensive study that, through large-scale experiments, demonstrates the extent to which current state-of-the-art AI Scientist systems can improve upon existing papers from top conferences. Specifically, they built a system called “ScientistTwo”, which conducts automated research through novel, detailed experimental designs (such as subset experiments and full experiments) as well as mechanisms like the Rebuttal Agent. As a result, they succeeded in improving the performance of existing papers in 86 out of 107 papers.
Critical strengths
-
S1. I think it is impressive that the experiment was conducted on such a large scale. While existing papers generally extend their methods to cover 10 or fewer papers, this paper extends it to 100 papers and demonstrates higher effectiveness.
-
S2. I found the addition of the Rebuttal Agent to be particularly innovative. While there have been methods in the past that improved papers based on reviews, none had incorporated this into the experimental process itself.
-
S3. The paper includes detailed experimental evaluations. It is impressive not only that the review score has improved compared to existing AI-generated papers, but also that the effectiveness of the Rebuttal Agent has been verified and an integrity audit has been conducted.
Critical weaknesses
I highly evaluate the abundant experimental results, research findings, and the design details of the framework presented in this paper. However, my concerns include the following:
- W1. Unclear Differences from and Novelty Relative to Existing Research
This paper, primarily in the Abstract and Introduction, describes systematic ablation studies, multi-metric evaluations, and improvements over existing top-conference research as if they were new contributions. However, these have already been achieved in existing research such as AI Scientist-v2 and Jr. AI Scientist [1] for ablation, and DeepScientist and Jr. AI Scientist [1] for improvements over existing top-conference papers. It is necessary to clearly articulate the unique technical novelty of ScientistTwo without underestimating the contributions of prior research.
- W2. Concerns Regarding Exaggeration of Scientific Claims and Hallucinations
Although the Integrity Audit verifies the consistency between experimental results and the code, the extent to which exaggeration of novelty, overgeneralization of experimental results, and hallucinations occur has not been sufficiently analyzed. In particular, further investigation is needed into whether repeated paper revisions by the Rebuttal Agent might exacerbate these issues.
- W3. Insufficient Evaluation of the Scientific Validity of the Research Methodology Itself
While this paper primarily evaluates performance improvements, it has not sufficiently verified whether the generated research methodology is scientifically valid or truly valuable as research. It is important not only to improve scores but also to evaluate the scientific reliability of the proposed methodology itself.
[1] Miyai+, Jr. AI Scientist and Its Risk Report: Autonomous Scientific Exploration from a Baseline Paper, TMLR2026
Questions for authors
Refer to Critical Weaknesses.
5 comments
[W3] Insufficient Evaluation of the Scientific Validity of the Research Methodology Itself
We appreciate the reviewer’s thoughtful feedback regarding the distinction between empirical benchmark gains and fundamental scientific validity. We fully agree that higher performance metrics alone do not guarantee sound research methodology, which is precisely why our evaluation framework explicitly decoupled empirical outcomes from methodological rigor.
Rather than relying solely on performance gains, we evaluated scientific validity of ScientistTwo’s proposed methods through dedicated, blinded human expert evaluations. As presented in Table 3 of our main manuscript, human experts evaluated the generated research across six core scientific dimensions, explicitly rating Methodological Soundness (averaging 4.3 / 5.0) and Scientific Maturity (averaging 4.1 / 5.0). Crucially, human evaluators examined whether the underlying hypotheses were theoretically motivated, whether the experimental designs adhered to rigorous scientific control standards, and whether the algorithmic contributions avoided trivial shortcuts or conceptual flaws.
To complement human evaluation at scale, we evaluated our generated papers against published baselines using both ScholarPeer and the Stanford Agentic Reviewer–AI reviewers that audits technical soundness, experimental validity, academic values, etc. Moreover, we would like to note that both ScholarPeer and the Stanford Agentic Reviewer demonstrate significant alignment with human review: ScholarPeer achieves a state-of-the-art Spearman rank correlation of 0.43 with human reviewers [1], while the Stanford Agentic Reviewer achieves 0.42 with human scores–exceeding 0.41, a typical human-to-human inter-annotator agreement [2] and predicts paper acceptance with an AUC of 0.75. This empirical baseline supports the premise that these simulated review scores serve as meaningful, non-random proxies for scientific quality. Using these AI reviewers, as shown in Table 1 of our manuscript, ScientistTwo-generated papers exceeded the average scores of accepted human-written manuscripts from ICLR 2026 and NeurIPS 2025.
[1] Goyal et al., ScholarPeer: A Context-Aware Multi-Agent Framework for Automated Peer Review, 2026
[2] Jiang and Ng, Tech Overview, papperreview.ai/tech-overview
Further Discussion Before the Deadline
Dear Reviewer 2dq5,
Thank you for your time and efforts again in reviewing our paper.
We kindly remind that the discussion period will end soon.
We believe that we sincerely and successfully address your comments, with the results of the supporting experiments.
If you have any further concerns or questions, please do not hesitate to let us know.
Thank you very much!
Authors
Deer Reviewer 2dq5
We deeply appreciate your invaluable time and efforts in reviewing our manuscript. We respond to each of your comments one-by-one in what follows. The revised manuscript has been marked in red.
[W1] Unclear Differences from and Novelty Relative to Existing Research
We thank the reviewer for the feedback. We agree that our Abstract and Introduction should more precisely highlight our specific technical novelties. We will thoroughly revise the Introduction and Related Work to properly cite the prior frameworks while clearly isolating ScientistTwo’s novel system architecture.
Specifically, first, while DeepScientist [1] formulates discovery as Bayesian Optimization over a single target objective, ScientistTwo evaluates and selects ideas jointly across multi-dataset, multi-metric benchmarks. Second, complementing DeepScientist’s surrogate-guided progressive filtering, ScientistTwo pairs subset-first screening with an engineering loop–tuning promising ideas on representative slices before scaling compute to full benchmarks, preventing brute-force resource waste and significantly improving the efficiency.
Third, we extend both how ablations guide discovery and how peer-review critiques are resolved. While AI Scientist-v2 [2], Jr. AI Scientist [3], and DeepScientist execute systematic ablations, they primarily run them after selecting a candidate method to analyze and tabulate component effects for paper writing. ScientistTwo turns ablation into a closed-loop refinement step: an Ablation Critic checks component-level changes to prune unhelpful modules, and re-engineer the method before drafting (see Table 8 of the main manuscript for the qualitative result). Finally, beyond automated review scoring or text-only reflection, our Rebuttal Agent writes and runs targeted supplementary experiments before paper finalization. Notably, while Jr. AI Scientist explicitly notes that expanding comparative methods (selecting and accurately reproducing baselines) remains beyond current autonomous AI Scientists and still requires human intervention, our Rebuttal Agent autonomously introduces and runs new baselines and held-out datasets without human input (e.g., see pages 69–73 for the example). Overseen by a Meta-Review Agent alongside automated integrity audits, this also directly mitigates the specification-gaming and citation risks highlighted in Jr. AI Scientist.
Together, these mechanisms represent a transition from code-and-write pipelines to an adaptive, self-auditing research agent. We explicitly clarified these distinctions and cited Jr. AI Scientist additionally, in the revised manuscript (see Section 2).
[1] Weng et al., DeepScientist: Advancing Frontier-Pushing Scientific Findings Progressively, 2026.
[2] Yamada et al., The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search, 2025.
[3] Miyai et al., Jr. AI Scientist and Its Risk Report: Autonomous Scientific Exploration from a Baseline Paper, 2026.
[W2] Concerns Regarding Exaggeration of Scientific Claims and Hallucinations
We thank the reviewer for this insightful comment regarding the critical need to guard against exaggerated claims and overgeneralization during autonomous paper generation and rebuttal.
First of all, we would like to clarify that the Rebuttal Agent's workflow is naturally designed to prevent exaggeration during paper revisions; its primary function is conditioned on execution feedback, where it autonomously designs, executes, and integrates targeted supplementary experiments to address critique. Because any textual revisions must directly reflect new empirical outputs verified by our execution logs, the iterative rebuttal cycle does not amplify ungrounded claims or hallucinated gains. Moreover, as shown in Table 5 of our manuscript, it is worth noting that the paper generated by ScientistTwo faithfully and accurately describes the codebase implementation and confirms that the methodology section is well explained without exaggeration. In addition, ScientistTwo is also guaranteed that the bibliography contains no hallucinated citations.
Second, to further alleviate your concern, we conducted a targeted audit evaluating: (1) novelty claim, and (2) whether experimental results are overgeneralized. Across all 33 papers evaluated on problem setups drawn from NeurIPS 2025, only two papers were flagged for mildly overgeneralized framing or overstated novelty. This confirms an empirically low rate of exaggeration (~6%) under the current pipeline.
We fully agree that systematically auditing text-level semantic integrity alongside code-level numerical consistency is essential for trustworthy scientific agents. We included these discussions and experiments in the revised manuscript (see Appendix B.3).