acceptodds
Withdrawn from ICLR 2027

AI-Driven Reproducibility and Analytical Robustness in Scientific Discovery

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reproducible and analytically robust research findings are essential to scientific progress, but evaluating these properties typically requires costly manual effort. We investigate whether LLM agents can help automate reproduction and robustness analysis, using a medical case study grounded in the Surveillance, Epidemiology, and End Results (SEER) database, a widely used U.S. cancer registry underlying tens of thousands of published studies. First, we introduce **SEER-Repro**, a benchmark of 20 published SEER-based papers together with the data and resources required to reproduce their findings. Second, we create an AI agent that successfully reproduces 79.1% of 2,190 atomic claims, while providing executable baselines for further analysis. Third, we develop an agentic analysis pipeline to test the robustness of the medical findings by identifying analytical choices underlying each finding, proposing defensible alternatives, and rerunning the analysis to determine whether those choices affect the conclusion. Across published papers in SEER-Repro, our pipeline finds that major effects on principal conclusions are flagged in 5.9% of fully reproduced findings. These results show how LLM agents can support scientific review by reproducing findings, systematically testing their robustness, and directing expert attention toward potentially consequential analytical choices.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Scientific discovery, Scientific reproduction, Analytical robustness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ttn6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a new benchmark, SEER-Repro, of 20 published SEER studies with an AI pipeline to evaluate if a paper is reproducible and robust to analytical choices. The proposed AI pipeline is tested against the benchmark, in which is achieves 79.1% of atomic claims from the papers and flags 34 of the 579 fully reproduced findings for prioritized review.

Critical strengths

None

Critical weaknesses
  1. This paper falls foul of what it aims to use AI to detect that is it is not reproducible. The paper stated Reproducibility ensures that research findings can be independently verified, however for this work that is not possible. The paper mainly utilises close source (proprietary) models to perform the analysis: GPT-5.6 Sol, GPT-5.6 Sol xhigh, Opus 4.8 max, Codex, Claude Code with a high reliance on GPT-5.6 Sol. The use of proprietary models makes this work inherently irreproducible. Proprietary model providers can update settings such as changing the system prompt, internal temperature, deprecate models and make changes to the model as they please. Which means the work cannot be independently verified or effectively reproduced.

  2. There is limited verification or validation of results. For the EXPERT ANALYSIS only one expert is used who is a coauthor with medical expertise which limits the verification of the findings, it is expected that independent analysis is performed with several experts to insure stability and consensus on the findings. It is unclear how the 10 papers for analysis where selected. Furthermore, what the inter-model agreement was for scoring models between 7–9, for example did some models score it 9 and others 1? Additionally, several prompts (Prompt 4, 24, 28, 32 from a cursory search) include statements such as do NOT fetch population data from the web. however no analysis is provided to check if this request was adhered to or not. This additionally extends to further do not statements. Furthermore, there is no analysis of how robust the findings are to changes in the prompts provided, or details on how these prompts where arrived at.

Questions for authors

See weaknesses but more concretely:

  1. Could the authors rerun all analysis with open source models? Detailing the exact evaluation settings, such that the results are reproducible.

  2. Could the authors add additional experts to perform EXPERT ANALYSIS?

  3. Can the authors provide the inter-model agreement was for scoring models between 7–9?

  4. Could you provide clarification on the verification procedure used to ensure the models adhered to the prompts.

  5. Could the authors explain how the prompts where arrived at, and provide analysis on the robustness of the prompts?

Reviewer_gM2uOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

An AI coding agent tried to replicate analyses from 20 published studies based on a US cancer registry, with each paper and its reported numbers in front of it. Another LLM marked each reported figure as reproducible/not with 5% tolerance. For those findings that were fully reproduced, six AI agents then changed one analysis choice at a time (for example, how the cancer site is coded) and an AI judge scored how much each change mattered. A minority of findings were flagged as possibly changing. An expert co-author checked ten flags: nine did not hold up and one is unresolved.

Critical strengths

Registry studies are used heavily and rarely rechecked, because recomputing them by hand is slow. The paper shows that one agent can recompute most of the reported numbers in several such studies, promising for larger scale deployment. The idea for testing robustness by varying analysis choices is sound.

Critical weaknesses
  1. The title, abstract and Introduction claim more than the results show. The abstract says "major effects on principal conclusions are flagged in 5.9% of fully reproduced findings", but what was measured is that an LLM rated some alternative 7–9 and a second LLM call linked it to a finding.

  2. Neither headline number is validated as a measurement. Both come from LLM instruments, run once, with no human comparison or reruns. Most numbers are reported without confidence intervals despite nondeterminism.

  3. How others can obtain the benchmark is not explained.

  4. The paper is harder to read than it need be, weighed down by unnecessary coined terms ("atomic claims", "consequential", "Qualification"), repetition of key statistics. The closest prior work is missing: RWE-bench (Li et al., arXiv 2603.22767), where agents reproduce observational studies on a clinical database.

Questions for authors

Were the resources and prompts developed on the same 20 papers that are evaluated; were any papers held out?

Reviewer_NMZLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper construct SEER-Repro, the first benchmark consisting of 20 original recent or highly cited papers published on the SEER data. It then formulates formulates the task of analytical robustness evaluation.

Critical strengths
  • This paper provides useful resources for a timely problem.
  • Expert examination of the results provides a good sanity check of the results, but it also seems to suggest that other results are questionable.
Critical weaknesses
  • It would be useful to situate the work in existing literature and similar efforts are going on in many areas.
  • The novelty is limited. That said, it is a nice documentation of results in this subset.
Questions for authors

Other related work includes: Veritas: Towards a General-Purpose Replication Tool for Scientific Research by Liu et al. ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers? by Ye et al.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.