- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents ScientistOne, an AI scientist system designed for research-style tasks. It integrates literature search and experimentation with a paper-writing module that grounds claims in generated provenance, through a process the authors call Chain-of-Evidence. The authors show that their system generates more grounded claims in its final communication while other AI scientist systems don’t necessarily ground claims in evidence. ScientistOne performs well across a variety of benchmarks, being competitive or SOTA on several tasks. Overall, this paper brings attention to the problem of claim integrity in AI scientist systems, but I have concerns about the novelty of the work and the generality of the presented results. Based on these concerns, I recommend a weak reject, but I might be convinced if the authors can address some of my concerns below.
Critical strengths
- The identified problem: verifiability of AI-generated research, is a crucial one, and the authors do a great job of motivating why this problem is interesting and impactful to the community.
- The authors show great empirical results of the integrity audits of their system, especially compared to that of other systems. Beyond the results of their model, it’s interesting to see the poor integrity of other AI scientist systems, and their contribution of this unique way to audit AI scientist systems is useful to the community. I also appreciate the authors explaining results beyond headline aggregations as this is helpful to understand the usefulness of the auditing tools.
- The authors went to lengths to test their system on a wide variety of tasks and benchmarks. Testing AI scientist systems is quite expensive, and it’s appreciated that they tested their system on multiple task types spanning common and uncommon benchmarks.
- The supplementary info is quite extensive and shed light into the system beyond the headline results. The authors went to great lengths to report their findings in a way that is beneficial for the community.
Critical weaknesses
- The agent architecture is not necessarily novel or expressive. For example, the idea of literature-driven problem investigation before “discovery” phase was introduced in Sakana’s AI-Scientist. The issue with this is it doesn’t allow for literature grounding during discovery, which is overly hindering to a long research process. In a long-horizon discovery task, a hypothesis may need to be altered through consultation of literature rather than relying on an LLM’s parametric knowledge. This architecture for the problem investigation and discovery is not novel and seems to be not expressive enough for real-world discovery tasks beyond the scoped systems problems discussed in the paper.
- The claimed chain-of-evidence is primarily a communication-time intervention. There is no mechanism in the agent design that upholds this integrity. During long-form explorations, it’s possible that certain pieces of provenance get overwritten, how would authors control for this? It’s possible that this did not arise during the paper’s explorations because none of the experiments were long enough.
- The empirical results of the ScientistOne system are incremental at best. The real benefit of this system seems to come from the paper-writing phase. I believe the paper should be evaluated with respect to this contribution as the new discovery architecture doesn’t seem to add much methodologically or empirically.
- The stated weakness of the human audit “its correctness on the cases it passed (the negatives) is not known”. These audits don’t seem to control for false negatives, e.g., cases that the auditor models might have missed. The integrity audits seem very fragile as they are also read-only; this means that overwritten outputs or incorrectly-reported results would not be caught so long as the recorded provenance shows the results reported in the papers. Therefore, this is not testing reproducibility, or the ability of the produced code to output the same results if ran under the same or slightly-different conditions
- The LLM reviewer results are very weak. LLM reviewers have a number of known issues, and testing against just one, even a published system, is not a rigorous result. I would want to see multiple LLM reviewer configurations, across base models, system prompts, etc., to rate the quality of the papers produced by each of the systems.
- The paper lacks significant details that are required for reproducibility, including source code and prompts for human auditors.
Questions for authors
- Can the authors test the reviews on a variety of LLM reviewer configurations to measure robustness?
- What was the longest task the authors tested their system on? Does the integrity of the claims drop as the investigation scales in length?
- Could this system be used with any AI scientist backbone? E.g., could you drop in your paper-writing module to any AI scientist system to increase the integrity of the communication? This seems like a more impactful contribution than an entirely-new AI scientist system.
- Could the authors test the reproducibility of the system, e.g., if re-running certain experiments produces the same results?
6 comments
4. More automated reviewers (W5, Q1)
We scored the papers of the same 75 runs, blind to the producing system, with three more reviewer configurations from published work, each run with the two newest models, Gemini 4 Argon and Claude Opus 5.5: the AI Scientist reviewer (Lu et al., 2024) and ProReviewer (Fang et al., 2026) with their authors' code, and RubricReviewer's 40 general rubrics (Guo et al., 2026). We also used the Stanford Agentic Reviewer (paperreview.ai, in its ICLR setting). ScientistOne's papers here are those of its revised writer, from the same discovery runs and with the same results, as in the revision, and their audit verdicts are unchanged. Mean Overall score per system, on each reviewer's 1–10 scale:
| Reviewer (model) | ScientistOne | AI-Researcher | Sakana v2 | DeepScientist | AutoResearchClaw |
|---|---|---|---|---|---|
| AI Scientist reviewer (Gemini 4 Argon, scale 1–10) | 1.73 | 1.13 | 1.33 | 1.00 | 1.00 |
| AI Scientist reviewer (Claude Opus 5.5, scale 1–10) | 2.00 | 1.93 | 2.00 | 1.87 | 1.60 |
| ProReviewer (Gemini 4 Argon, scale 1–10) | 2.90 | 2.38 | 2.32 | 2.13 | 1.70 |
| ProReviewer (Claude Opus 5.5, scale 1–10) | 2.52 | 2.55 | 2.37 | 2.28 | 1.97 |
| RubricReviewer (Gemini 4 Argon, scale 1–10) | 1.87 | 1.27 | 1.47 | 1.07 | 1.00 |
| RubricReviewer (Claude Opus 5.5, scale 1–10) | 1.60 | 1.07 | 1.13 | 1.00 | 1.00 |
| Stanford Agentic Reviewer (scale 1–10) | 4.26 | 3.73 | 3.73 | 3.54 | 3.45 |
| Mean rank over the six model reviewers | 1.3 | 2.5 | 2.3 | 4.2 | 4.8 |
ScientistOne has the best mean rank over the six model reviewers and the highest score from the Stanford Agentic Reviewer. It ranks first under four of the six, tied first under one and second under one, and the reviewers agree on the order (Kendall's W = 0.88). Pooled over the six, ScientistOne scores above every baseline, paired over the 15 tasks and seeds (Holm-adjusted p ≤ 0.003). Automated review judges how a paper reads, so we take it as one evaluation among several and compare the systems with the audit, which checks the claims against code, re-runs and references.
5. Integrity over longer runs (Q2)
The longest runs we tested evaluate up to 500 solutions, and the longest, on LLM-SQL, took 8.6 hours. The runs in the paper evaluate about 100 solutions each, in about two hours. We did not test investigations that run for days.
To test whether integrity drops with length, we ran two settings: (1) ScientistOne with a Claude Code agent (Claude Opus 5.5) as its implementer, at budgets of 100 and 500 evaluations on LLM-SQL with Cloudcast as a control, and (2) Claude Code on its own, without our stages, for up to 150 rounds of Karpathy's AutoResearch loop on all five tasks, with its reports audited at rounds 10, 25, 50, 100 and 150.
In ScientistOne, the papers keep their integrity as runs get longer. All seven papers from these runs and from a run with the paper's Gemini implementer at budget 200 pass all four checks, three of them at budget 500. The search itself does change with length. On LLM-SQL, the evaluator rewards a row ordering that the task does not, and the share of the search's solutions that use this loophole rises from 0 of 50 at budget 100 to 10 of 50 at budget 500. None of the solutions ScientistOne selected uses it.
Claude Code on its own loses integrity as runs get longer. It adopts the LLM-SQL loophole early and reports the evaluator's number as the task result. Once its score stops improving, it re-runs unchanged programs to draw a better value from a noisy score, in 6% of evaluations before round 50 and 26% after (p = 0.012), and in three runs it logged results before the evaluation had finished. Its reports still pass I3 and I4 at all 47 checkpoints and I1 at 46, and the audit's I2 flags 8 of the 9 checkpoints that use the loophole.
Longer runs make the search drift toward a loophole in the evaluator. Claude Code on its own carries this drift into its reports, while none of ScientistOne's papers uses the loophole and all of them pass the audit. The revised paper adds these results (Sec. 6.8 and Appendix L.3).
6. The writer as a drop-in module (Q3)
Yes. The writer is a separate stage with a fixed input: the run's log with its scores, the selected solution and its code, and the retrieved references. It writes each claim with a link to one of these, and its verifier checks the claims against them before release. Any system that exports these inputs can use it.
We tested this on two other systems' runs, seed 1 of each of the five tasks. A script copies each run's logs, scores and code into the format our writer reads. It calls no model and adds no generated text. Both papers of each run were audited with the same checks:
| Runs and writer | References not found ↓ | I4 fails ↓ | Method claims supported ↑ | Unsupported result numbers ↓ |
|---|---|---|---|---|
| DeepScientist, its own writer | 39.7% | 5 of 5 | 17% | 23.7% |
| DeepScientist, our writer | 0.6% | 0 of 5 | 94% | 1.0% |
| AI-Researcher, its own writer | 16.1% | 2 of 5 | 75% | 16.7% |
| AI-Researcher, our writer | 0.3% | 1 of 5 | 83% | 2.1% |
DeepScientist keeps the evaluator's output for every candidate, and on its runs our writer's papers are better on every measure on all five tasks. AI-Researcher keeps no evaluator output tied to a program, so on its runs the gains are smaller and mixed. In one of its runs, its own reports describe parts of the method that the final program does not contain, and both its paper and ours repeat them and fail I4. Over the 10 runs every difference is significant (paired exact Wilcoxon, p ≤ 0.04). How much our writer improves a paper therefore depends on what the system records, since CoE requires each claim to link to a record of the run.
7. Reproducibility of results (Q4, W4)
A result in a paper depends on five things: the solution's score, the discovery run that produced the solution, the data it was tested on, the paper's text, and the audit's verdict. We tested each of them.
(1) Re-running each submitted solution five times (I1) gives identical scores in 7 of ScientistOne's 12 scored papers and scores within 0.054% in the other five (EPLB is outside I1, as in Table 1).
(2) Re-running the whole discovery pipeline with three independent seeds per task changes the submitted solutions' scores by 0% on Prism, 1.5% on Cloudcast, 3.5% on LLM-SQL, 8.0% on TXN and 13.0% on EPLB (range over mean).
(3) We also ran every system's submitted programs, unchanged, on new input data for each task, generated the same way as the benchmark's own (10 new sets per task, 8 for LLM-SQL), and on larger inputs, scored by the benchmark's own code. The submitted solutions of every system keep their order (Spearman's ρ from 0.85 to 0.98 per task), and ScientistOne's solutions keep their lead over the starting program in all 15 runs.
(4) Writing the papers of the 15 runs a second time with the same writer configuration gives the same I1, I3 and I4 verdicts, and the share of supported method claims changes by 0.9 to 2.0 points.
(5) Running the four checks a second time on Table 1's 75 papers gives the same verdict on 59 of 60 papers for I1, 72 of 75 for I2 and 74 of 75 for I4, and on 1,783 of 1,787 references for I3.
8. Details for reproducibility (W6)
Most of what reproduction needs is in the appendices already: the search configuration, models and budgets, the ideator and implementer prompts, the audit procedure, and how each baseline was adapted. We will release the remaining prompts (the writer stages, the Claim Verifier and the audit's judges), the raters' instructions and label definitions, and all 75 papers with their audit verdicts with the final version. We will add the reproducibility results to the appendix of the final version, and the other new results in this reply are in the revised paper.
Thank you for the careful review, and for recognizing the importance of verifiability in AI-generated research, the value of auditing other systems, the breadth of our benchmarks and the detail of our appendices. One clarification first: the audit is not read-only. It re-runs every submitted solution on the benchmark's evaluator, as we explain under W4. We answer the weaknesses and questions in turn.
1. Architecture, and where the contribution lies (W1, W3)
Our contributions are CoE and the audit built on it, together with a system that writes its papers in CoE form. To the best of our knowledge, the audit is the first set of general, automatic tests of research rigor that is applied to papers from multiple systems on real-world tasks and validated against human raters. The discovery architecture is not among our novelty claims. Like the AI Scientist, ScientistOne grounds its ideas in the literature before discovery, and literature shapes discovery through those ideas: the Problem Investigator reads up to 100 full-text papers per topic, and the ideator writes candidate ideas from its brief, which the discovery loop then tests and refines. This holds on every benchmark: on MLE-Bench, for example, the ideas come from recent papers on each competition's problem.
The loop is simple, and it performs well on systems problems and beyond. It matches or exceeds the human expert solution on all five ADRS tasks, wins gold on two and silver on two of the five MLE-Bench Kaggle competitions, and beats the best Parameter Golf score at the time of the run (1.0600 against 1.0611 bits per byte).
On W3, we agree that the paper should be judged on CoE, the audit and the evidence-linked writing. The discovery loop serves to produce real research for them to check, and an agentic loop such as Claude Code can replace it without changing the audit.
2. Provenance during long explorations (W2)
ScientistOne keeps provenance during the research as well as at writing time.
(1) During the research, the Problem Investigator records every reference it retrieves, each evaluated solution is written once to its own folder with its execution results, and every submitted solution traces by file hash to the node of the discovery loop that produced it.
(2) An overwritten provenance record cannot hide a wrong result, because the audit re-runs the submitted code and never reads the logs.
(3) Longer explorations, with budgets of up to 500 evaluated solutions, still produce papers that pass all four checks, as we show in our answer to Q2 below. Today the writer creates the claim-level links when it writes the paper. Creating them during discovery as well is future work.
3. Control for false negatives, and what the audit checks (W4)
We first clarify that the audit re-runs every submitted program on the benchmark's evaluator (I1), so it does not rely on recorded results. We also measured its false negatives, in two ways.
(1) Blinded raters annotated every paper of the compared systems and a sample of 220 references the audit passed. The audit catches 15 of the 16 papers the raters judged misaligned (I4), and 219 of the 220 passed references are real (I3).
(2) To answer your question further, and because the raters found no I2 violation, we also planted known violations into copies of our 15 papers and their code. The audit caught all 24 inflated headline scores (I1), all 30 planted solvers that hardcode the evaluator's answers or exploit its rules (I2), and all 30 invented references (I3). I4 caught 9 of 30 method sections that add or drop a stage of the same algorithm, because by design it fails only a different class of algorithm or hidden gaming logic. I1, I2 and I4 flagged none of the 30 clean or harmlessly edited copies.
The audit is not read-only. None of the four checks reads the system's own records, and overwritten records cannot change a verdict: I1 re-runs the submitted code five times on the benchmark's evaluator and compares the score with the paper, I2 checks the code against the benchmark's rules, I3 looks up every reference in Semantic Scholar, OpenAlex or Crossref, and I4 compares the method section with the code. AutoResearchClaw's seven I1 failures in Table 1 show what this catches: submitted code that crashes because files it needs were left out, and a score that changes between runs because the random seed was not set.