- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes a new benchmark, SEER-Repro, of 20 published SEER studies with an AI pipeline to evaluate if a paper is reproducible and robust to analytical choices. The proposed AI pipeline is tested against the benchmark, in which is achieves 79.1% of atomic claims from the papers and flags 34 of the 579 fully reproduced findings for prioritized review.
Critical strengths
None
Critical weaknesses
-
This paper falls foul of what it aims to use AI to detect that is it is not reproducible. The paper stated
Reproducibility ensures that research findings can be independently verified, however for this work that is not possible. The paper mainly utilises close source (proprietary) models to perform the analysis:GPT-5.6 Sol,GPT-5.6 Sol xhigh,Opus 4.8 max,Codex,Claude Codewith a high reliance onGPT-5.6 Sol. The use of proprietary models makes this work inherently irreproducible. Proprietary model providers can update settings such as changing the system prompt, internal temperature, deprecate models and make changes to the model as they please. Which means the work cannot beindependently verifiedor effectively reproduced. -
There is limited verification or validation of results. For the
EXPERT ANALYSISonly one expert is used who is acoauthor with medical expertisewhich limits the verification of the findings, it is expected that independent analysis is performed with several experts to insure stability and consensus on the findings. It is unclear how the 10 papers for analysis where selected. Furthermore, what the inter-model agreement was for scoring models between 7–9, for example did some models score it 9 and others 1? Additionally, several prompts (Prompt 4, 24, 28, 32 from a cursory search) include statements such asdo NOT fetch population data from the web.however no analysis is provided to check if this request was adhered to or not. This additionally extends to furtherdo notstatements. Furthermore, there is no analysis of how robust the findings are to changes in the prompts provided, or details on how these prompts where arrived at.
Questions for authors
See weaknesses but more concretely:
-
Could the authors rerun all analysis with open source models? Detailing the exact evaluation settings, such that the results are reproducible.
-
Could the authors add additional experts to perform
EXPERT ANALYSIS? -
Can the authors provide the inter-model agreement was for scoring models between 7–9?
-
Could you provide clarification on the verification procedure used to ensure the models adhered to the prompts.
-
Could the authors explain how the prompts where arrived at, and provide analysis on the robustness of the prompts?