acceptodds
Under review as a conference paper at ICLR 2027

ScientistOne: Verifiable Autonomous Research via Chain-of-Evidence

Abstract

Autonomous research agents now produce competitive solutions and polished manuscripts, but their outputs can carry verifiability failures that presentation-focused evaluation misses: fabricated citations, unreproducible scores, and method descriptions that diverge from the code. These failures share a root: today's autonomous research systems are not built to trace their claims back to evidence. We address this with three contributions. First, Chain-of-Evidence (CoE), a verifiability framework under which every claim traces to its evidence source. Second, ScientistOne, an end-to-end autonomous research system that maintains evidence chains by construction across literature review, solution discovery, and paper writing. Third, CoE Integrity Audit, a post-hoc audit whose four integrity checks (score verification, specification violation, reference verification, and method–code alignment) apply the same criteria across systems. Across 75 papers from five systems on five frontier systems-research tasks, every baseline shows at least one systematic failure mode. Among the three baselines in the comparison, up to 24% of references are hallucinated, score verification passes in as few as 42% of papers, and method–code alignment ranges from 20% to 87%. ScientistOne is best or tied for best on all four checks, with none of its 340 distinct references hallucinated, 12/12 scores verified, and 14/15 method sections that match the code, while matching or exceeding the human expert solution on all five tasks. Against blinded human raters, the audit is right on all 48 score extractions and 219 of 220 passed references, and its method–code verdicts agree with the raters about as often as the raters agree with each other (92%). Our findings show that a deliberately simple discovery loop already reaches expert-level solutions, and we argue that the bottleneck in autonomous research is verifiability: CoE enables a paper to be audited before it is trusted, by both humans and machines.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
autonomous research, AI for science, Chain-of-Evidence, verifiability, research integrity, provenance, LLM agents, paper writing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_u15oOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ScientistOne, an AI scientist system designed for research-style tasks. It integrates literature search and experimentation with a paper-writing module that grounds claims in generated provenance, through a process the authors call Chain-of-Evidence. The authors show that their system generates more grounded claims in its final communication while other AI scientist systems don’t necessarily ground claims in evidence. ScientistOne performs well across a variety of benchmarks, being competitive or SOTA on several tasks. Overall, this paper brings attention to the problem of claim integrity in AI scientist systems, but I have concerns about the novelty of the work and the generality of the presented results. Based on these concerns, I recommend a weak reject, but I might be convinced if the authors can address some of my concerns below.

Critical strengths
  1. The identified problem: verifiability of AI-generated research, is a crucial one, and the authors do a great job of motivating why this problem is interesting and impactful to the community.
  2. The authors show great empirical results of the integrity audits of their system, especially compared to that of other systems. Beyond the results of their model, it’s interesting to see the poor integrity of other AI scientist systems, and their contribution of this unique way to audit AI scientist systems is useful to the community. I also appreciate the authors explaining results beyond headline aggregations as this is helpful to understand the usefulness of the auditing tools.
  3. The authors went to lengths to test their system on a wide variety of tasks and benchmarks. Testing AI scientist systems is quite expensive, and it’s appreciated that they tested their system on multiple task types spanning common and uncommon benchmarks.
  4. The supplementary info is quite extensive and shed light into the system beyond the headline results. The authors went to great lengths to report their findings in a way that is beneficial for the community.
Critical weaknesses
  1. The agent architecture is not necessarily novel or expressive. For example, the idea of literature-driven problem investigation before “discovery” phase was introduced in Sakana’s AI-Scientist. The issue with this is it doesn’t allow for literature grounding during discovery, which is overly hindering to a long research process. In a long-horizon discovery task, a hypothesis may need to be altered through consultation of literature rather than relying on an LLM’s parametric knowledge. This architecture for the problem investigation and discovery is not novel and seems to be not expressive enough for real-world discovery tasks beyond the scoped systems problems discussed in the paper.
  2. The claimed chain-of-evidence is primarily a communication-time intervention. There is no mechanism in the agent design that upholds this integrity. During long-form explorations, it’s possible that certain pieces of provenance get overwritten, how would authors control for this? It’s possible that this did not arise during the paper’s explorations because none of the experiments were long enough.
  3. The empirical results of the ScientistOne system are incremental at best. The real benefit of this system seems to come from the paper-writing phase. I believe the paper should be evaluated with respect to this contribution as the new discovery architecture doesn’t seem to add much methodologically or empirically.
  4. The stated weakness of the human audit “its correctness on the cases it passed (the negatives) is not known”. These audits don’t seem to control for false negatives, e.g., cases that the auditor models might have missed. The integrity audits seem very fragile as they are also read-only; this means that overwritten outputs or incorrectly-reported results would not be caught so long as the recorded provenance shows the results reported in the papers. Therefore, this is not testing reproducibility, or the ability of the produced code to output the same results if ran under the same or slightly-different conditions
  5. The LLM reviewer results are very weak. LLM reviewers have a number of known issues, and testing against just one, even a published system, is not a rigorous result. I would want to see multiple LLM reviewer configurations, across base models, system prompts, etc., to rate the quality of the papers produced by each of the systems.
  6. The paper lacks significant details that are required for reproducibility, including source code and prompts for human auditors.
Questions for authors
  1. Can the authors test the reviews on a variety of LLM reviewer configurations to measure robustness?
  2. What was the longest task the authors tested their system on? Does the integrity of the claims drop as the investigation scales in length?
  3. Could this system be used with any AI scientist backbone? E.g., could you drop in your paper-writing module to any AI scientist system to increase the integrity of the communication? This seems like a more impactful contribution than an entirely-new AI scientist system.
  4. Could the authors test the reproducibility of the system, e.g., if re-running certain experiments produces the same results?
6 comments

Thank you for the careful review, and for recognizing the importance of verifiability in AI-generated research, the value of auditing other systems, the breadth of our benchmarks and the detail of our appendices. One clarification first: the audit is not read-only. It re-runs every submitted solution on the benchmark's evaluator, as we explain under W4. We answer the weaknesses and questions in turn.

1. Architecture, and where the contribution lies (W1, W3)

Our contributions are CoE and the audit built on it, together with a system that writes its papers in CoE form. To the best of our knowledge, the audit is the first set of general, automatic tests of research rigor that is applied to papers from multiple systems on real-world tasks and validated against human raters. The discovery architecture is not among our novelty claims. Like the AI Scientist, ScientistOne grounds its ideas in the literature before discovery, and literature shapes discovery through those ideas: the Problem Investigator reads up to 100 full-text papers per topic, and the ideator writes candidate ideas from its brief, which the discovery loop then tests and refines. This holds on every benchmark: on MLE-Bench, for example, the ideas come from recent papers on each competition's problem.

The loop is simple, and it performs well on systems problems and beyond. It matches or exceeds the human expert solution on all five ADRS tasks, wins gold on two and silver on two of the five MLE-Bench Kaggle competitions, and beats the best Parameter Golf score at the time of the run (1.0600 against 1.0611 bits per byte).

On W3, we agree that the paper should be judged on CoE, the audit and the evidence-linked writing. The discovery loop serves to produce real research for them to check, and an agentic loop such as Claude Code can replace it without changing the audit.

2. Provenance during long explorations (W2)

ScientistOne keeps provenance during the research as well as at writing time.

(1) During the research, the Problem Investigator records every reference it retrieves, each evaluated solution is written once to its own folder with its execution results, and every submitted solution traces by file hash to the node of the discovery loop that produced it.

(2) An overwritten provenance record cannot hide a wrong result, because the audit re-runs the submitted code and never reads the logs.

(3) Longer explorations, with budgets of up to 500 evaluated solutions, still produce papers that pass all four checks, as we show in our answer to Q2 below. Today the writer creates the claim-level links when it writes the paper. Creating them during discovery as well is future work.

3. Control for false negatives, and what the audit checks (W4)

We first clarify that the audit re-runs every submitted program on the benchmark's evaluator (I1), so it does not rely on recorded results. We also measured its false negatives, in two ways.

(1) Blinded raters annotated every paper of the compared systems and a sample of 220 references the audit passed. The audit catches 15 of the 16 papers the raters judged misaligned (I4), and 219 of the 220 passed references are real (I3).

(2) To answer your question further, and because the raters found no I2 violation, we also planted known violations into copies of our 15 papers and their code. The audit caught all 24 inflated headline scores (I1), all 30 planted solvers that hardcode the evaluator's answers or exploit its rules (I2), and all 30 invented references (I3). I4 caught 9 of 30 method sections that add or drop a stage of the same algorithm, because by design it fails only a different class of algorithm or hidden gaming logic. I1, I2 and I4 flagged none of the 30 clean or harmlessly edited copies.

The audit is not read-only. None of the four checks reads the system's own records, and overwritten records cannot change a verdict: I1 re-runs the submitted code five times on the benchmark's evaluator and compares the score with the paper, I2 checks the code against the benchmark's rules, I3 looks up every reference in Semantic Scholar, OpenAlex or Crossref, and I4 compares the method section with the code. AutoResearchClaw's seven I1 failures in Table 1 show what this catches: submitted code that crashes because files it needs were left out, and a score that changes between runs because the random seed was not set.

4. More automated reviewers (W5, Q1)

We scored the papers of the same 75 runs, blind to the producing system, with three more reviewer configurations from published work, each run with the two newest models, Gemini 4 Argon and Claude Opus 5.5: the AI Scientist reviewer (Lu et al., 2024) and ProReviewer (Fang et al., 2026) with their authors' code, and RubricReviewer's 40 general rubrics (Guo et al., 2026). We also used the Stanford Agentic Reviewer (paperreview.ai, in its ICLR setting). ScientistOne's papers here are those of its revised writer, from the same discovery runs and with the same results, as in the revision, and their audit verdicts are unchanged. Mean Overall score per system, on each reviewer's 1–10 scale:

Reviewer (model)ScientistOneAI-ResearcherSakana v2DeepScientistAutoResearchClaw
AI Scientist reviewer (Gemini 4 Argon, scale 1–10)1.731.131.331.001.00
AI Scientist reviewer (Claude Opus 5.5, scale 1–10)2.001.932.001.871.60
ProReviewer (Gemini 4 Argon, scale 1–10)2.902.382.322.131.70
ProReviewer (Claude Opus 5.5, scale 1–10)2.522.552.372.281.97
RubricReviewer (Gemini 4 Argon, scale 1–10)1.871.271.471.071.00
RubricReviewer (Claude Opus 5.5, scale 1–10)1.601.071.131.001.00
Stanford Agentic Reviewer (scale 1–10)4.263.733.733.543.45
Mean rank over the six model reviewers1.32.52.34.24.8

ScientistOne has the best mean rank over the six model reviewers and the highest score from the Stanford Agentic Reviewer. It ranks first under four of the six, tied first under one and second under one, and the reviewers agree on the order (Kendall's W = 0.88). Pooled over the six, ScientistOne scores above every baseline, paired over the 15 tasks and seeds (Holm-adjusted p ≤ 0.003). Automated review judges how a paper reads, so we take it as one evaluation among several and compare the systems with the audit, which checks the claims against code, re-runs and references.

5. Integrity over longer runs (Q2)

The longest runs we tested evaluate up to 500 solutions, and the longest, on LLM-SQL, took 8.6 hours. The runs in the paper evaluate about 100 solutions each, in about two hours. We did not test investigations that run for days.

To test whether integrity drops with length, we ran two settings: (1) ScientistOne with a Claude Code agent (Claude Opus 5.5) as its implementer, at budgets of 100 and 500 evaluations on LLM-SQL with Cloudcast as a control, and (2) Claude Code on its own, without our stages, for up to 150 rounds of Karpathy's AutoResearch loop on all five tasks, with its reports audited at rounds 10, 25, 50, 100 and 150.

In ScientistOne, the papers keep their integrity as runs get longer. All seven papers from these runs and from a run with the paper's Gemini implementer at budget 200 pass all four checks, three of them at budget 500. The search itself does change with length. On LLM-SQL, the evaluator rewards a row ordering that the task does not, and the share of the search's solutions that use this loophole rises from 0 of 50 at budget 100 to 10 of 50 at budget 500. None of the solutions ScientistOne selected uses it.

Claude Code on its own loses integrity as runs get longer. It adopts the LLM-SQL loophole early and reports the evaluator's number as the task result. Once its score stops improving, it re-runs unchanged programs to draw a better value from a noisy score, in 6% of evaluations before round 50 and 26% after (p = 0.012), and in three runs it logged results before the evaluation had finished. Its reports still pass I3 and I4 at all 47 checkpoints and I1 at 46, and the audit's I2 flags 8 of the 9 checkpoints that use the loophole.

Longer runs make the search drift toward a loophole in the evaluator. Claude Code on its own carries this drift into its reports, while none of ScientistOne's papers uses the loophole and all of them pass the audit. The revised paper adds these results (Sec. 6.8 and Appendix L.3).

6. The writer as a drop-in module (Q3)

Yes. The writer is a separate stage with a fixed input: the run's log with its scores, the selected solution and its code, and the retrieved references. It writes each claim with a link to one of these, and its verifier checks the claims against them before release. Any system that exports these inputs can use it.

We tested this on two other systems' runs, seed 1 of each of the five tasks. A script copies each run's logs, scores and code into the format our writer reads. It calls no model and adds no generated text. Both papers of each run were audited with the same checks:

Runs and writerReferences not found ↓I4 fails ↓Method claims supported ↑Unsupported result numbers ↓
DeepScientist, its own writer39.7%5 of 517%23.7%
DeepScientist, our writer0.6%0 of 594%1.0%
AI-Researcher, its own writer16.1%2 of 575%16.7%
AI-Researcher, our writer0.3%1 of 583%2.1%

DeepScientist keeps the evaluator's output for every candidate, and on its runs our writer's papers are better on every measure on all five tasks. AI-Researcher keeps no evaluator output tied to a program, so on its runs the gains are smaller and mixed. In one of its runs, its own reports describe parts of the method that the final program does not contain, and both its paper and ours repeat them and fail I4. Over the 10 runs every difference is significant (paired exact Wilcoxon, p ≤ 0.04). How much our writer improves a paper therefore depends on what the system records, since CoE requires each claim to link to a record of the run.

7. Reproducibility of results (Q4, W4)

A result in a paper depends on five things: the solution's score, the discovery run that produced the solution, the data it was tested on, the paper's text, and the audit's verdict. We tested each of them.

(1) Re-running each submitted solution five times (I1) gives identical scores in 7 of ScientistOne's 12 scored papers and scores within 0.054% in the other five (EPLB is outside I1, as in Table 1).

(2) Re-running the whole discovery pipeline with three independent seeds per task changes the submitted solutions' scores by 0% on Prism, 1.5% on Cloudcast, 3.5% on LLM-SQL, 8.0% on TXN and 13.0% on EPLB (range over mean).

(3) We also ran every system's submitted programs, unchanged, on new input data for each task, generated the same way as the benchmark's own (10 new sets per task, 8 for LLM-SQL), and on larger inputs, scored by the benchmark's own code. The submitted solutions of every system keep their order (Spearman's ρ from 0.85 to 0.98 per task), and ScientistOne's solutions keep their lead over the starting program in all 15 runs.

(4) Writing the papers of the 15 runs a second time with the same writer configuration gives the same I1, I3 and I4 verdicts, and the share of supported method claims changes by 0.9 to 2.0 points.

(5) Running the four checks a second time on Table 1's 75 papers gives the same verdict on 59 of 60 papers for I1, 72 of 75 for I2 and 74 of 75 for I4, and on 1,783 of 1,787 references for I3.

8. Details for reproducibility (W6)

Most of what reproduction needs is in the appendices already: the search configuration, models and budgets, the ideator and implementer prompts, the audit procedure, and how each baseline was adapted. We will release the remaining prompts (the writer stages, the Claim Verifier and the audit's judges), the raters' instructions and label definitions, and all 75 papers with their audit verdicts with the final version. We will add the reproducibility results to the appendix of the final version, and the other new results in this reply are in the revised paper.

Reviewer_n7LLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Chain-of-Evidence (CoE), a framework linking research claims to supporting artifacts; ScientistOne, an autonomous research pipeline incorporating these links; and an integrity audit covering score reproducibility, specification compliance, reference existence, and method–code alignment. The evaluation examines generated papers on five ADRS tasks using a common backbone, compares their artifact integrity, and assesses audit judgments through blinded human evaluation. Additional benchmarks evaluate solution discovery beyond ADRS. The reported results demonstrate that competitive solutions can coexist with unreliable research artifacts.

Critical strengths
  • The comparative evaluation examines consequential failures against their underlying artifacts. Re-running solutions and comparing manuscripts with code and bibliographic records provides substantially more informative evidence than automated reviewer scores alone. Using a shared backbone reduces an important confound, and the concrete failure analyses make the findings actionable.

  • The audit receives meaningful validation and transparent error analysis. The blinded human evaluation, examination of passed references, and disclosure of disagreements, false positives, missed errors, and adaptation problems strengthen confidence in the bounded findings. The authors also candidly report inconclusive ablations. These are substantive merits of the empirical study.

Critical weaknesses

AI polished rewrite of my weaknesses:

  1. The incremental contribution beyond established provenance-preserving generation and artifact-aware auditing is insufficiently established. The general diagnosis and proposed direction have close precedents. Section 2 acknowledges claim-level auditability, evidence-grounded artifacts, and validation collapse, but does not adequately distinguish the substantive requirement or mechanism introduced by CoE. Two particularly relevant predecessors are missing. data-to-paper already implements programmatic backward links from manuscript numerical results through intermediate artifacts to generating code and data. Luo et al. already empirically demonstrate that auditing workflow logs and code reveals methodological failures missed by manuscript-only assessment. Their controlled failure examples differ from the naturally generated outputs studied here, but directly precede the central motivation and auditing approach. Ifargan et al., NEJM AI, 2025; Luo et al., 2025, §6. Moreover, Wang et al., already cited, propose claim packages, evidence traces, validator provenance, and validation entering the research loop as a validity condition. The relationship to these design prescriptions deserves substantive treatment. Wang et al., §6. These precedents do not subsume the present comparative study. Nevertheless, they substantially narrow the conceptual novelty of the framework and its motivating diagnosis. The strongest contribution appears to be the integrated audit and comparative empirical findings. I am not yet convinced that the paper identifies a sufficiently substantial advance beyond applying established principles and checks to additional systems.

  2. The evidence leaves the distinctive benefit and operational scope of CoE unresolved. The CoE principle requires faithful evidential support at every link. The implemented checks cover only part of that requirement. The common audit excludes citation support and conclusion validity; internally, citation support is assessed against abstracts, while methodological claims are checked through textual overlap with experimental logs (Appendix K.6). I appreciate that Sections 3 and 7 acknowledge these limitations. However, the abstract and introduction’s language about every claim and evidence chains maintained “by construction” suggests broader assurance than the evaluated checks establish. The experiments also leave attribution uncertain. Table 1 compares complete pipelines differing in retrieval, writing context, orchestration, and verification. Table 5 establishes an important retrieval benefit, but does not demonstrate improved measured claim precision from the final Claim Verifier: the reported rates are 88.6% with it and 90.6% without it. This ablation retains earlier grounding and critique stages and has limited statistical power, so it does not establish that CoE or verification is ineffective. It nevertheless leaves the final verifier’s incremental benefit unresolved. The provenance record is potentially valuable in its own right, but its advantage for downstream audit effort, coverage, or error detection is not measured. Finally, the categorical conclusion that discovery capability is no longer the bottleneck exceeds evidence from these scoped benchmark tasks. The additional MLE-Bench and Parameter Golf experiments evaluate discovery without evaluating the full audit, so they do not broaden the evidence for the central verifiability contribution. My concern is whether the proposed advance is sufficiently established. Narrowing the headline claims would improve accuracy, but would still leave the contribution and significance question above.

My further comments (unpolished):

  • I personally feel the abstract is long and difficult to read with all these numbers and claims. Can you shorten it and remove numbers as much as possible?

  • I believe you are not the first paper to find evidence-chain failures, in fact there is some more literature about it which you should cite I believe, see below?

  • Same about fact that enforcing artifact structures is necessary for avoiding hacking towards evidence chain failures as other papers claim.

  • I'm questioning the novelty of this paper with respect to uncited literature from the periods of March-May 2026 (which are not considered competing work for ICLR standards).

Papers mentioned (citation from different ChatGPT chats):

  • Ifargan et al. (2025). Autonomous LLM-Driven Research—from Data to Human-Verifiable Research Papers. NEJM AI.
  • Luo, Kasirzadeh, and Shah (2025). The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systems. arXiv v2.
  • Wang et al. (2026). Workflow Closure Is Not Scientific Closure in Auto-Research Systems. arXiv v1. Already cited by ScientistOne; used for comparison.
  • Another workshop paper/survey (Not sure, probably. not for citation) claiming similar things: "Agent Systems for Academic Research Automation", ICML 2026 AI for Science Workshop.
  • pAI/MSc: ML theory research with humans on the loop; M Abdelmoneum, P Beneventano, T Poggio; arXiv preprint arXiv:2604.20622
  • Messeri and Crockett (2024). Artificial intelligence and illusions of understanding in scientific research. Nature. Epistemic context.
Questions for authors
  • I understand that this paper was written months ago when the community thought that hand-designing agentic scaffoldings was a key innovation towards authomating long-horizon tasks. I wonder what the authors think is now the value of such contributions. I have the perception that the community moved on from this task as they think that dynamic orchestration, say codex with GTP-6 Astra, is good enough for most academic research (sub?)tasks within a research project. My belief is that, either fortunately or unfortunately, the community thinks that executing a well-scoped research work IS already automated. I wonder what the authors think about it.

  • Also, did the author read pAI/MSc (Abdelmoneum et al., March 2026) mentioned above? What do they think? In my human perception the two papers are very similar and it is my understanding that most contribution of this paper are already in that one--although with a slightly different philosophical framing. I think it's necessary that you properly compare to that paper in terms of chain of evidence and for the actual agentic system proposed. It would also be nice to see how you compare to them in terms of performance (say, in table 1). What precise assurance does a passing audit establish? Maybe you could distinguish mechanically enforced properties, fallible semantic judgments, and assumptions about the adequacy of the evidence. The broader distinction between technical productivity and scientific understanding discussed by Messeri and Crockett is also relevant context -- which I think the competing work as pai/msc doesn't cite as well, so I may be wrong about the relationship.

  • I did not understand to what extent you make the problem "verifiable" in the paper above (pai/msc) the authors argue that research execution is not verifiable end-to-end not even by giving artifact contracts as in your CoE. Can you compare to that claim and motivate any difference?

  • What specific requirement, mechanism, or empirical insight distinguishes CoE from the closest prior approaches? Please separate the contribution of the framework, the generation pipeline, and the comparative audit, particularly relative to data-to-paper, Luo et al., and the substantive design proposals in the work already cited.

  • Can existing records demonstrate the distinctive benefit of the evidence-handling machinery? For example, which unsupported claims were corrected, or how much does retaining native provenance improve subsequent auditing compared with reconstructing evidence from ordinary artifacts? I recognize the included coding-agent baseline; the remaining question is which mechanisms account for the observed differences. Also, I remember talking with collegues arguing there is no benefit in practice, do you genuinely have any idea of in what settings there are no benefits?

I'm very willing to change my mind if the questions and weaknesses are properly addressed.

5 comments

We thank the reviewer for the detailed reading and for the papers they pointed us to. The comments helped us state more precisely how CoE differs from prior work, and the questions on what a passing audit establishes and which mechanisms matter led to new analyses. We uploaded a revised paper, with the changes in blue. We also appreciate the reviewer's recognition of our comparison against the underlying artifacts and of the human validation of the audit.

1. Comparison with prior and concurrent work (Polished W1, Q2, Q4, Unpolished W2 to W4)

Table R1 compares ScientistOne with the related work. The revised paper cites all of these papers in Sec. 2 and includes this table in Appendix L. I1 to I4 are our four audit checks: score reproducibility (I1), reward hacking (I2), hallucinated references (I3) and method mis-description (I4). ✓ means implemented and evaluated, – partial or limited to the work's own outputs, and × absent. The number counts the systems audited.

WorkTypeClaim–evidence frameworkEnd-to-end systemAudits other systemsValidated by human ratersI1I2I3I4
data-to-paper, Ifargan et al. (2025)System–✓××××–×
Hidden Pitfalls, Luo et al. (2025)Study××✓ 2–×–×–
pAI/MSc, Abdelmoneum et al. (2026)System–✓××××××
VMA, Beneventano et al. (2026)Position–×××××××
ARA, Liu et al. (2026)Format✓×××–××–
ResearchArena, Zhang et al. (2026)Study×–✓ 3––×✓–
Workflow Closure, Wang et al. (2026)Position–×–×××××
ScientistOne (ours)System + audit✓✓✓ 5✓✓✓✓✓

data-to-paper (Ifargan et al., NEJM AI 2025) is a pioneering study that we should have cited. The revised paper cites it and notes that ScientistOne's writer follows a similar design. Both build traceability into the generation pipeline. data-to-paper links the numbers in its Results section to the code that produced them, and CoE adds links from method statements to the code and from citations to scholarly records. data-to-paper also evaluated only its own papers, by hand, while our audit checks finished papers from five systems automatically and is validated against blinded human raters.

Hidden Pitfalls (Luo et al., 2025) shares our motivation: an auditor that reads logs and code detects pitfalls more accurately than one that reads only the paper (82% against 55%). It studies four experimental-design pitfalls in two systems on a synthetic task. Our audit adds three checks (I1, I3, I4) and applies them to papers that five systems produced on real tasks.

pAI/MSc (Abdelmoneum et al., 2026) keeps a pipeline's state in inspectable artifacts with validation gates between stages, and VMA (Beneventano et al., 2026) proposes a claim-level bundle of claims, evidence objects and verifier types, close to CoE. Neither reports results that test whether its design works.

Workflow Closure (Wang et al., 2026) is a position paper that prescribes tying claims to their evidence and validating them inside the research loop. We implement claim–evidence links in a system and measure their effects with an audit applied to five systems.

Concurrent work (Unpolished W2 to W4). ResearchArena (Zhang et al.) finds fabricated results, paper–code mismatches and fake references in 117 agent-written papers by inspecting their code and logs by hand, and proposes no remedy. ARA (Liu et al.) and ResearchLoop (Xia and Wang) build claim-to-evidence links into their own artifacts and check only their own outputs. Our audit tests finished papers from other systems.

2. What is new: the framework, the pipeline and the audit (Q4)

To the best of our knowledge, our study is the first to propose general, automatic tests of research rigor, apply them to papers from multiple systems on real-world tasks, and validate their reliability against human raters. As the reviewer asks, we separate our three contributions.

(1) The framework. CoE requires numbers, method statements and citations to link to evidence that can check them: a run's records, its code and scholarly records. data-to-paper links only the numbers in its Results section to code.

(2) The generation pipeline. ScientistOne writes its papers in CoE form, end to end with no human in the loop, and unlike data-to-paper and pAI/MSc it was compared with four other systems on the same tasks.

(3) The comparative audit. Four automatic checks (I1 to I4), validated against blinded human raters, show that systems with competitive scores still write papers whose claims do not hold: up to 24% of references are hallucinated, up to 7 of 12 reported scores do not reproduce, and method–code alignment ranges from 20% to 87%. The concurrent ResearchArena reports such failures from a manual audit, while our checks run automatically.

Empirical insights into what makes a paper pass. (1) The run's records decide it. With them, both our writer and PaperOrchestra pass the four checks on every paper. With only the research plan and the final score, our writer's method descriptions match the code 28% of the time, compared with 94% with the records (Sec. 6.5). (2) The same holds for other systems. Given the records of DeepScientist's runs, our writer's papers have 1.0% unsupported result numbers, compared with 23.7% in DeepScientist's own papers, and 94% of their method claims are supported by the code, compared with 17%. AI-Researcher keeps no evaluator output for its candidates, and on its runs our writer gains little.

3. The value of hand-designed scaffolds when agents are strong (Q1)

To answer this question, we replaced our discovery loop with Claude Code running on its own (Claude Opus 5.5, up to 150 rounds). It finds strong solutions, matching or exceeding ScientistOne's on four of the five tasks.

(1) The agent sometimes logged a score before running the evaluation. We instructed it to keep a structured workspace, and it did, but it still logged 4 scores before evaluating, one of them invented. It corrected each within seconds, but nothing outside the agent ensures it. Because the agent writes its own record, structure alone does not make the record faithful.

(2) Our scaffold keeps the record faithful by isolating evaluation and scoring from the agent. In ScientistOne, the agent writes code, and the pipeline runs the evaluators and records the scores itself, so an early or invented score cannot reach the paper. Our audit does not catch logging problems, since its four checks compare the final paper with the final code.

(3) Keeping the record is also cheaper than re-running. A re-run can be very costly: retraining a model from scratch, for example, can take days or weeks. On ADRS, checking result numbers against the record costs about 3% as much as rebuilding them.

We believe the value of hand-designed scaffolds lies in preserving provenance, which makes research results auditable without re-running the research.

4. Comparison with pAI/MSc, and what a passing audit establishes (Q2)

We read pAI/MSc carefully, and the revised paper cites it as concurrent work. Both systems keep a research pipeline's state in files, and both separate structural checks from scientific validity.

(1) The two systems have different scopes. pAI/MSc supports ML theory research with humans able to steer it, while ScientistOne runs empirical CS tasks end to end.

(2) The two check different things. pAI/MSc's artifact contract checks that each stage's named outputs exist and are well-formed, and its traceability check links theorem labels to its own claim graph. CoE links each number, method statement and citation in a paper to evidence that can check it, and our audit checks them against a re-run, the code and scholarly records. In other words, pAI/MSc's checks target theory research, and ours check a paper against its artifacts.

(3) There is also a practical reason that pAI/MSc cannot be compared in Table 1: its experiments run through AI-Scientist-v2's tree search, the engine of the Sakana baseline in Table 1, which needed the most adaptation of all baselines and still gave incomplete results on ADRS.

A passing audit gives two kinds of assurance and depends on one assumption.

  • Mechanically enforced. The audit re-runs the submitted code on the benchmark's own evaluator and gets the paper's headline score (I1), and it finds every reference in Semantic Scholar, OpenAlex or Crossref (I3). Of the 220 references the audit passed, 219 are real publications.

  • Fallible semantic judgments. An LLM judge finds no violation of the benchmark's rules (I2) and no mismatch between the method section and the code (I4). On I4 it agrees with blinded human raters at κ = 0.82.

  • Assumptions about the adequacy of the evidence. The audit assumes that the benchmark's evaluator measures what the task intends, and that the evidence behind a claim is enough to support it. It does not judge whether the experiments are sufficient, such as the number of seeds or the strength of the baselines.

We also tested the audit on violations we planted ourselves, one per copy of our papers and their code: inflated headline scores, solvers that hardcode the evaluator's answers or exploit its rules, invented references, and method sections that add or drop a stage. I1 caught all 24 inflated scores, I2 all 30 planted solvers, and I3 all 30 invented references. I4 caught 9 of the 30 stage-level edits: by design it accepts embellishment and omitted detail, and fails a paper only when the described algorithm is of a different class or hides gaming logic. I1, I2 and I4 flagged none of the 30 clean or harmlessly edited copies.

We leave two checks to future work: whether a cited paper's full text supports the sentence citing it, and whether the conclusions follow from the results. A pass says nothing about understanding. Messeri and Crockett separate that risk from errors and failures of reproducibility, and the audit addresses only the second kind. The revised paper adds the planted-violation test (Sec. 6.7).

5. What is verifiable, against pAI/MSc's argument (Q3)

Verifiability depends on the claim. We separate three levels.

  • Evidence-bound claims, such as reported scores, references and method descriptions, can be checked against a re-run, academic search engines and the code.
  • Inferential claims, such as whether a citation supports its sentence or a conclusion follows from the results, can be judged but not checked mechanically.
  • The value of the research, its novelty, significance or theoretical correctness, has no automatic verifier. pAI/MSc's argument that end-to-end research value cannot be scored by an automatic reward concerns the third level, and we agree with it.

ScientistOne targets the first level: in closed-loop computational research the system has full access to code, logs and a benchmark evaluator, so these claims can be verified, and our audit measures this across five systems. CoE also defines requirements for the second level, and we leave their evaluation to future work. This is where CoE differs from an artifact contract such as pAI/MSc's (Abdelmoneum et al., 2026), whose validation gates check that the expected intermediate outputs exist and are well-formed, while CoE checks each claim against its evidence.

Reviewer_fpWFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is based on the assumption that the key problem for today’s autonomous research systems is that they are not built to trace their claims back to evidence. With this in mind, the paper produced three artifacts:

(1) Chain-of-Evidence (CoE), a verifiability framework under which every claim traces to its evidence source.

(2) ScientistOne, an end-to-end autonomous research system that maintains evidence chains by construction across literature review, solution discovery, and paper writing.

(3) An automated CoE Integrity Audit, a post-hoc audit whose four integrity checks (score verification, specification violation, reference verification, and method–code alignment) apply the same criteria across systems.

ScientistOne is compared to 5 autoresearch systems on 5 benchmarks and across 75 papers. It performs better than these baseline systems on the four verfiability tests studied.

The CoE Integrity Audit is evaluated via a human study and shows 92% agreement with the human annotators/raters (although this is just a raw agreement score not a standard interannotator agreement measure that takes random agreement into account).

Critical strengths

The paper tackles important problems in the area of autoresearch. It's focus on claim verifiability is a nice driving force. It is impressive that the work includes both the development and evaluation of an autoreserach system AND an autoresearch auditing system.

Critical weaknesses

The paper's motivation is more general than what the paper actually investigates and accomplishes. It seems initially that the paper will study "veriability failures" in general in the context of automatically generated research papers, checking that "EVERY CLAIM (emphasis added) traces to its evidence source (line 017 and elsewhere). The paper gives three examples of these errors ("fabricated citations, unreproducible scores, and method descriptions that diverge from the code" lines 13-14). But it is just these three types of verifibility failures that are addessed (...actually three are mentioned in the abstract, four types appear in figure 1 and the introduction). As stated throughout the paper then, these claims/contributions are overstated.

In the same vein, the proposed system, ScientistOne, is specifically designed to correct these three/four types of failures, so it is not surprising that it performs better with respect to these failures than systems that are not specificially targeting these failures. The paper should be up front about this.

The first contribution (081), the Chain-of-Evidence claim taxonomy + evidence should be qualified. The paper itself references Zhang et al. 2026, which identifies 9 types of "claims", including those developed here.

The degree to which the second and third contributions are novel (082-086) is also not made clear: there are multiple similar systems that exist as evidenced in the paper's related work section.

The paper's conclusions that "the bottleneck in autonomous research is verifiability" and "[the paper's] CoE [framework] enables a paper to be audited before it is trusted, by both humans and machines" seem to be similarly over-reaching.

The paper assumes a claim-based framework but does not explain how claims of the four stated types are identified and distiguished from the many other types of claims that a paper makes.

Questions for authors
  1. 030-031 In this sentence "Our findings show that a deliberately simple discovery loop already reaches expert-level solutions", what is the "discovery loop" that is mentioned? I don't think it's been described in the abstract or the introduction. Is it referring to ScientistOne? The CoE Integrity Audit? What is the "solution"? And how is it determined to exhibit the level of an expert?

  2. Does the "we" in line 053 refer to the authors (or other human auditors) or to the automated system/framework, CoE"?

    053 "we audited 75 papers from five autonomous research systems on five benchmark tasks"

Figure 1 implies it is CoE, but CoE seems to be only looking for four types of verifiability failures while the paper reads as though these are the ONLY types of verifibility failures that appear across the 75 papers. The paper further references Zhang et al. 2026 (line 074) to back up CoE's focus on these four types of failures, but Zhang et al. performed a MANUAL audit and identified nine characterized nine types of failures. I think that this type of writing is misleading.

  1. 049-051 citations needed. "the evaluations commonly adopted in the community, automated review scores and benchmark leaderboards, measure how a paper reads and whether the scores are competitive, and neither checks whether individual claims trace to supporting evidence."

  2. 053 (and in the abstract) "we audited 75 papers from five autonomous research systems on five benchmark tasks" --> "we audited 75 papers PRODUCED BY five autonomous research systems TO TACKLE THE SAME five benchmark tasks"

  3. 212-213 It does not seem enough to verify citation claims agains just the cited paper's abstract.

  4. Do each of the 75 papers involve all 5 of the frontiers systems and all 5 of the research tasks? The mapping among these isn't clear.

4 comments

We thank the Reviewer for the careful review and constructive comments. We agree that the paper's claims were stated more broadly than what our four checks actually test, and we tightened the abstract, introduction and conclusion in the revised paper, which we uploaded with the changes in blue.

1. "Every claim" overreaches the four checks (line 017 and elsewhere) (W1)

We agree. Section 3 states Chain-of-Evidence as a general principle, and the audit tests four parts of it: score verification, specification violation, reference existence and method–code alignment. CoE also defines requirements for citation support and conclusions, which this study does not yet evaluate (ScientistOne's writer runs a preliminary citation check against the cited abstracts while writing, see our answer to Q4). The abstract and introduction promised more than the four checks establish. In the revised paper, we scoped them to the claims the audit checks in this study, and Sec. 7 states what the checks do not cover. The abstract also gave three failure examples (lines 012–014) where Figure 1 and Section 1 evaluate four, and the revised abstract gives all four.

2. ScientistOne is designed for these failures, so its lead is unsurprising (W2)

First, CoE is a general framework that targets the rigor of the whole paper. CoE defines evidence requirements for four claim types (numerical, methodological, citation and conclusion), and ScientistOne is built to meet them. A writer swap shows that whether a paper passes the checks does not depend on our writer's modules, but on the system design for provenance and auditing. Both our writer and the unmodified PaperOrchestra, the open-source writer ours was built from, pass all four checks with the full records. Without the records, both fail method–code alignment: ours on 13 of 15 papers, PaperOrchestra on 14 of 15 (Sec. 6.5).

Second, the audit is independent of the systems it evaluates. It checks a system's paper and code against the benchmark's evaluator and rules (I1, I2), academic search engines (I3) and each other (I4). We report audit results on ScientistOne and four open-source baseline systems, and each check is validated against blinded human raters. On I4 method–code alignment, the audit agrees with the raters on 92% of papers, a Cohen's κ of 0.82. ScientistOne's writer also checks things the audit does not, such as whether the paper overclaims and whether the abstract of each cited paper supports the sentence citing it.

Third, ScientistOne also does well on evaluations we did not design. It earns gold medals on two MLE-Bench tasks and the top Parameter Golf score at the time of the run. We also scored the papers of all 75 runs with more automated reviewers from published work, and ScientistOne's papers score highest (Sec. 6.4).

3. Qualify the taxonomy against Zhang et al. 2026 (lines 073–075, 081) (W3)

Zhang et al. do not define a claim taxonomy. Their nine items (Sec. 3.4) are the scoring dimensions of a review rubric (novelty, soundness, significance, clarity, reproducibility, experimental rigor, references, reference integrity, results integrity). Their manual audit (Sec. 5.2) reports three failure modes in experimental rigor (fabricated results, underpowered experiments and plan/execution mismatch), together with fabricated references. Fabricated results overlap with our I1, plan/execution mismatch with our I4, and fabricated references with our I3. The revised paper drops the citation at line 074 and states in Sec. 2 which of these failure modes our checks cover.

CoE covers more than these failure modes. Zhang et al. find failures by inspecting finished papers by hand. CoE is a requirement on how a paper is built: every numerical, methodological, citation and conclusion claim traces to its evidence. ScientistOne's writer links every numerical claim to a log line, and 627 of 639 match it. The I1-I4 audit checks the parts of CoE that can be verified automatically on papers from other systems. Underpowered experiments are a matter of experimental design: a claim can trace to its evidence, but that evidence can still be too weak to support the claim. The revised Sec. 7 names statistical power among what the checks leave out.

We stated the scope of Contribution 1 precisely in Sec. 1 and Sec. 3: the four claim categories (numerical, methodological, citation and conclusion) are a taxonomy of research claims that can be traced to artifacts, and they do not cover all scientific claims.

4. Novelty of ScientistOne and CoE Integrity Audit (lines 082–086) (W4)

Our contribution is a unified framework that makes research produced by AI systems auditable against the rigor standards CoE defines, and correctable where it falls short of them. To our knowledge, two parts of it are new. The first is an automatic audit that checks the claims of finished papers from different systems against re-runs, the code and scholarly records, validated against blinded human raters. Applied to five systems, it shows that systems with competitive scores still write papers whose claims do not hold. The second is a system that writes its papers in CoE form end to end, with numbers, method statements and citations linked to their evidence. It passes every check on 14 of its 15 papers. Some building blocks, such as scholarly API lookups, evaluator re-execution and LLM code inspection, have been studied in prior work. We sharpened lines 082–086 and Sec. 2 accordingly, and Table 18 in the revised paper (Appendix L.1) compares ScientistOne with the closest prior and concurrent systems.

  • ScientistOne (Contribution 2). Discovery systems on ADRS (AdaEvolve, EvoX, AlphaEvolve) optimize code without literature grounding or paper writing. End-to-end paper generators (AI Scientist-v2, AutoResearchClaw, DeepScientist, AI-Researcher) write standard LaTeX without linking claims to artifacts. The closest prior system, data-to-paper (Ifargan et al., 2025), links the numbers in its Results section to the code that produced them. ScientistOne also links method statements and citations to their sources, across all three stages.

  • CoE Integrity Audit (Contribution 3). Prior and concurrent integrity studies inspect papers by hand (data-to-paper on its own papers, Zhang et al., 2026, and Wang et al., 2026), propose auditability measures without applying them (Rasheed et al., 2026), or evaluate one check in isolation (MLR-Bench, Chen et al., 2025a, and SciCoQA, Baumgärtner & Gurevych, 2026). Our audit combines four checks (I1–I4) in one automated audit, at $0.49 and about 4 minutes per paper.

5. "The bottleneck is verifiability" and "CoE enables a paper to be audited before it is trusted" overreach (W5)

We agree that passing I1–I4 is necessary for trust but not sufficient. A paper can pass all four checks and still have weak baselines or unsupported conclusions. ScholarPeer rejected 9 of the 15 papers in our submission, and Sec. 7 leaves checks of conclusions to future work. We scoped the first statement to the five ADRS tasks in Sec. 7, and removed the second.

Within that scope, the audit is to our knowledge the first automatic comparison of AI research systems on properties of research rigor. Existing evaluations compare these systems by benchmark score or by automated review of their papers. The audit checks whether selected types of claims in their papers trace to evidence, with the same tasks and checks for every system. We believe more checks are needed to make AI research systems comparable on research rigor, and we state this in the revised Sec. 7.

6. How claims of the four types are identified and distinguished from other claims (W6)

We should have explained this more clearly in the paper. ScientistOne's writer tags every sentence that states a number, a comparison or a finding from another paper with the evidence for it: a record of the run, a citation key, or "unsourced" when it has none. The tag sets the claim type. A sentence tagged with a citation key is a citation claim. A sentence tagged with a run record is a numerical claim if it contains a number, and a methodological claim if it does not. The Claim Verifier checks each claim by the rule for its type and drops unsourced claims and claims with malformed tags. Overclaims in the framing are softened before the paper is written. These are the writer's own mechanisms for checking and correcting its claims. The I1–I4 audit does not use them: it reads only the finished paper, its code and its references. We added this explanation to Sec. 4.3, and Sec. 3 now states that the four types differ in the evidence they require.

7. "Discovery loop", "solution", and "expert-level" (lines 030–031) (Q0)

We apologize for using these terms before defining them. The revised Sec. 1 defines all three. We take them from prior work on AI-driven algorithm discovery, AlphaEvolve (Novikov et al., 2025) and ADRS (Cheng et al., 2025b):

  • Discovery loop: ScientistOne's second stage. We call it simple because it is a fixed parallel search that keeps the best branches, without the evolutionary machinery of AlphaEvolve, EvoX and AdaEvolve, such as program populations, evolved search strategies and adaptive compute scheduling.
  • Solution: the Python program that solves an ADRS task, such as GPU model placement for Prism or transaction scheduling for TXN.
  • Expert-level: the solution's score, re-evaluated on the same benchmark's evaluator, matches or exceeds the human expert baseline reported in ADRS. ScientistOne reaches this level on all five tasks.

8. "We" at line 053, and the Figure 1 / Zhang et al. framing (Q1)

We apologize for the unclear wording. "We" at line 053 means the authors, who ran the automated CoE Integrity Audit and then checked its flags by hand: an author read every flag the audit raised, and two blinded raters re-judged 60 papers and 312 references.

The audit is a different thing from CoE. CoE is a framework that guides how AI research systems should be built: it requires the claims in their papers to be supported by evidence and leaves the design to the system. The I1–I4 audit is an evaluation kit with four checks of research rigor. The 75 papers can have other failures, and Figure 1 blurs this distinction. We revised line 053 and the Figure 1 caption to name the four-check audit, and moved the discussion of Zhang et al. from line 074 to Sec. 2 (see our answer to W3).

9. Citations for lines 049–051 (Q2)

Thank you for the suggestion. We added these citations at lines 049–051:

  • Automated review scores: Lu et al. (2024) (The AI Scientist), Goyal et al. (2026) (ScholarPeer).
  • Benchmark leaderboards: Chan et al. (2024) (MLE-Bench), Huang et al. (2024) (MLAgentBench), Cheng et al. (2025b) (ADRS).
  • Manuscript-only review missing artifact-level failures: Zhang et al. (2026), and Chen et al. (2025a) (MLR-Bench) for mismatches between results and execution.

10. Rewrite of line 053 (Q3)

Thank you, we adopted the rewrite. Sec. 1 now reads "we ran our four-check audit on 75 papers produced by five autonomous research systems to tackle the same five benchmark tasks", and the abstract was revised the same way.

11. Verifying citations against only the abstract (lines 212–213) (Q4)

We agree that a citation is best checked against the cited paper's full text. ScientistOne's writer uses abstracts for two practical reasons. Reading the full text of every cited paper costs many more tokens, and the full text is often unavailable: many cited papers are behind paywalls, so a full-text check would cover some references and not others, which would make its results noisy. The abstract check still catches clear misattributions. In one of our papers, for example, it removed a sentence about conflicts between database transactions that cited a paper on CPU scheduling. The same access problem is why the I1–I4 audit has no check of citation support: I3 checks only that each reference exists, which needs no full text. We stated this limitation in Sec. 4.3, and Sec. 7 lists a full-text check of citation support as future work.

12. Mapping of 75 papers to systems and tasks (Q5)

We apologize that the paper did not state this mapping clearly. Each paper is one run of one system on one task with one seed: 5 systems × 5 tasks × 3 seeds = 75 papers, 15 per system and 3 per system and task. All systems used Gemini 3.1 Pro for both code and writing. The revised Abstract, Figure 1 caption and Sec. 1 state this mapping.

Thank you again for your review. We would appreciate it if you could take a look at our response and the additional results in the revised manuscript. We would be happy to answer any questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.