acceptodds
Under review as a conference paper at ICLR 2027

AN APPROACH TO QUANTIFYING THE IMPACT OF ARTIFICIAL INTELLIGENCE ON THE SCIENCES: PERSPECTIVES FROM BIOMEDICINE

Abstract

Machine learning (ML) is now ubiquitous in the biological sciences, and unless implemented carefully it is prone to data leakage and related modes of failure. Estimating how often this happens has so far required manual, domain-specific systematic reviews wynants2020prediction,roberts2021common,whalen2022navigating, which do not scale beyond a few hundred papers per study. Large language models (LLMs), combined with established taxonomies of ML failure modes kapoor2023leakage,lones2024avoid, make a literature-scale estimate tractable khraisha2024can. We applied a lightweight LLM pipeline, a non-ML comparison arm, and a sandboxed code-execution audit to biomedical papers retrieved from OpenAlex — of them classified as using ML — and followed every code-availability claim in the cohort to its source. Classification was checked against human review of a blind, stratified -paper validation set, on which the pipeline was accurate ( CI –), or (–) once review articles are recoded under the pre-specified rule. Within the ML corpus, of papers reported performance metrics inadequate to the claim they were offered in support of, and showed evidence of illegitimate features. Of papers claiming available code, () had a link that still resolved, released code matching the paper's own description of its methods, and exactly one could be built and executed in an isolated sandbox — where it did reproduce its reported result. ML papers nonetheless accrued more citations than non-ML papers after adjustment for publication year and subfield (incidence rate ratio , CI –) and appeared in venues with roughly the median impact factor, and papers whose availability claims failed the audit suffered no citation penalty. This is, to our knowledge, the first literature-scale estimate of ML implementation quality in biomedicine, and it suggests that ML is subject to hasty over-adoption there, with reproducibility and code availability routinely overstated.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
failure-analysis, metascience, ai-for-science

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Vx1POpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduced a pipeline to audit 27,200 biomedical papers published since 2011, with a particular focus on the 11,736 papers involving ML. The pipeline used LLMs to classify papers and screen for potential failure modes. The results were validated by rerunning the LLMs on a small subset of papers and conducting human reviews of a separate, smaller subset. The findings indicated that a strikingly high proportion of ML-related biomedical papers made claims that were not adequately supported.

Critical strengths
  • The paper introduces an LLM-assisted pipeline that classifies and audits biomedical papers.

  • The results are drawn from a large corpus (27,200 papers) that covers a long time span (2011-2016).

  • The (astonishing) findings of the ML-biomedical papers are supported by detailed statistical results.

Critical weaknesses
  • The paper can be hard to understand for readers who are not very familiar with a few papers. The most important one is the leakage taxonomy of Kapoor & Narayanan [1]. Given that the paper currently lacks a Related Works section, it is highly recommended that the authors add a section discussing prior work in the field and their methodologies, and provide background.

[1] Kapoor, Sayash, and Arvind Narayanan. "Leakage and the reproducibility crisis in machine-learning-based science." Patterns 4.9 (2023).

  • The manual validation process is not described in sufficient detail. Given the seriousness of the reported findings, which some readers may interpret as accusations, the authors should provide a more detailed account of the “accuracy control” procedure (Section 2.2) and the “Forensic Deep Read” process (Section 2.5). Please refer to the questions below for specific points requiring clarification.
Questions for authors

Dataset

  • What is the overall quality of the 27,200 biomedical papers included in the corpus? The authors mention recording “the two-year mean citedness (OpenAlex’s open JIF analogue) and h-index” for each paper’s publication source, but is there any relationship between citation impact and the validity of the reported claims? For example, are more highly cited papers less likely to make inadequately supported claims? Are papers published in venues with higher h-indices more likely to provide reliable, well-maintained codebases?

Human Reviews

  • In line 97, the authors mentioned there is a blind human review of 60 papers, and the results are reported in the paragraph starting from line 328. However, since there is only one human rater, it is unclear if the mismatch between human and LLM classification is because of reasonable subjectivity or model defects. Comparing the statistics between human raters' alignment with the statistics between humans and LLMs can help to answer the question. Additionally, a sample with 60 points drawn from 27,200 samples is too small. Even if all 60 labels are correct, the one-sided 95% upper confidence bound on the true error rate can still be very low.

  • Section 2.5 introduced a forensic deep read on 200 papers. It would be helpful to include case studies for those papers when discussing the results. Additionally, why would the LLM detector fail in some cases?

  • For all the human reviews, what is the background of the annotators? How to make sure the quality of the reviews is reasonable?

Other

The abstract on OpenReview contains "\citep{kapoor2023leakage,lones2024avoid}".

3 comments
AuthorsOpenReview ↗

Human Rater and Overall Quality

Hello! I've done some extra analysis.

  1. I audited 1000 papers by hand for quality, and checked this against 2 different frontier LLM audits of the whole corpus. Quality is mostly good, though some (~20%) papers that were non-English but on topic, related in topic but not empirical, or occasionally concatenated documents slipped through.
  2. I've just received confirmatory spot-checks from 3 more humans, backing the 60-paper review with 100 each. The human reviewers all have different technical backgrounds - I'm not really sure how to establish legitimacy without compromising anonymity! I appreciate your comment about case studies - however, I am very hesitant to include such a thing. As you accurately pointed out, these results may be read as accusations, which is not our intent! Our point is simply that the process of vetting and replicating papers is not working properly, since the vast majority don't include sufficient information. To include a specific example would turn a possible accusation into a definite and specific call-out.

Please include the results from the spot-checks from the additional 3 human annotators in the camera-ready version if the paper is accepted.

I am slightly surprised by the efficiency of auditing 1000 papers. Can you share more details regarding the auditing process? For example, how many minutes do you spend on each paper? Is there a certain standard procedure?

I will keep my previous rating.

Copied from my clarification for other reviewer:

This audit was simply a human-eyes check for the following classifications:

  1. Does not render
  2. Non-english language (Largely a non-issue, since all the papers in languages I could understand were actually on-topic, though I still ruled them as failing this check).
  3. Irrelevant (not empirical ML/biomedical research - this included a very few off-topic papers (ML in fluids engineering without an explicit biomedical connection, eg) and a fair number of papers about adoption or reception of ML tools in the clinic.
  4. Conference proceedings/concatenations where our review pipeline might have conflated multiple texts
  5. No issue

I used a click-through interface which displayed the pdf full text of the paper, making it fairly easily to audit each text in between a few seconds (for instance, a paper in Korean or Russian was ruled out in <5 seconds at a glance) to 10-15 minutes (rarely, to assess whether a paper was actually biomedical - fluids engineering, for instance). The vast majority of the papers were either obviously relevant - a straightforward english publication about applying ML in bioscience, or about non-ML based biomedical research, without any concatenation - or obviously failing. Anything in a foreign language or concatenated or clearly about policy/adoption of ML was very quick to rule out. That's not to say I didn't have to click through papers and rest my eyes for the better part of a few days of course! These were not audits in the sense of the pipeline, that's my mistake for using the same word. This 1000-paper check was to ensure that the labeling and filtering applied properly and the pool of papers wasn't artifically inflated, which would drive down the apparent reproducibility.

Reviewer_QJzDOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper performs a large-scale audit of papers in the biomedical sciences that use machine learning. The audit is conducted with the help of LLMs, and flags papers that fail a set of criteria taken from an existing taxonomy of failure modes, including insufficiently supported claims, missing code, etc. The paper also performs an analysis of the differential impact of ML-related vs non-ML related papers in biomedicine, quantified using citation counts and impact factors. The paper performs validation using cross-model-family validation, a separate human annotator, and a forensic deep read of a smaller subset of papers.

Critical strengths
  1. The size of the audit is an order of magnitude larger than previous audits of this kind, and it surfaces authoring-related recommendations (backed up by quantitative analyses) that can have a positive impact on biomedical research.
  2. The research seems to have been done with care, with multiple forms of validation performed.
  3. Beyond just flagging problematic ML-related papers, the additional analyses performed on impact using citations and impact factors should be of wide interest to biomedical researchers.
Critical weaknesses

While the paper was not too hard to follow, the prose is quite dense in parts and reads very much like LLM-generated or at least LLM-flavored prose, which is okay in small quantities but exhausting to read in large quantities.

The other large-ish limitation is the use of a single human annotator for the blind spot-check of papers, and the lack of a human-human agreement figure.

Questions for authors

No questions for the authors.

3 comments
AuthorsOpenReview ↗

Made writing style more concise, added parallel audits

Hello! I've been working to reduce the density of the writing. More importantly, I've done a hand audit of 1000 papers (backed by LLM audits of different random samples). I'll add the exact details and figure shortly, but the vast majority of the papers fit our filtering criteria. I also got back results from 3 more humans who backed the spot-check (100 papers each) to support classification accuracy against a human baseline.

Reviewer_QJzDOpenReview ↗

Response

Thanks for your response. Like reviewer Vx1P, I am also surprised that you were able to hand-audit 1000 papers between the time the reviews were released and the time you posted your comment (i.e., approximately 83 papers per day on average). Please share the details of your auditing process.

This audit was simply a human-eyes check for the following classifications:

  1. Does not render
  2. Non-english language (Largely a non-issue, since all the papers in languages I could understand were actually on-topic, though I still ruled them as failing this check).
  3. Irrelevant (not empirical ML/biomedical research - this included a very few off-topic papers (ML in fluids engineering without an explicit biomedical connection, eg) and a fair number of papers about adoption or reception of ML tools in the clinic.
  4. Conference proceedings/concatenations where our review pipeline might have conflated multiple texts
  5. No issue

I used a click-through interface which displayed the pdf full text of the paper, making it fairly easily to audit each text in between a few seconds (for instance, a paper in Korean or Russian was ruled out in <5 seconds at a glance) to 10-15 minutes (rarely, to assess whether a paper was actually biomedical - fluids engineering, for instance). The vast majority of the papers were either obviously relevant - a straightforward english publication about applying ML in bioscience, or about non-ML based biomedical research, without any concatenation - or obviously failing. Anything in a foreign language or concatenated or clearly about policy/adoption of ML was very quick to rule out. That's not to say I didn't have to click through papers and rest my eyes for the better part of a few days of course! These were not audits in the sense of the pipeline, that's my mistake for using the same word. This 1000-paper check was to ensure that the labeling and filtering applied properly and the pool of papers wasn't artifically inflated, which would drive down the apparent reproducibility.

Reviewer_56GDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The goal of this work is mainly to evaluate the claims from biomedical papers (that use ML in some form).

This paper uses an LLM pipeline (Gemini) over 27200 open-acess biomedical papers from OpenAlex. About 11.7k are classified as ML papers; each ML paper is screened for five failure modes from Kapoor & Narayanan’s leakage taxonomy.

Findings:

40.3% of ML papers use inadequate metrics for their claims, and 13.4% show illegitimate features. Of ~2,056 code claims, only one executes, and even that one doesn’t verify the headline result.

A 200-paper read shows that the cheap flags track inflated claims (16.5% vs. 4.3% median improvement) rather than leakage itself. The authors frame this as a negative result about their own instrument.

Critical strengths
  1. The number of papers this work covers is larger than that of most other similar analyses.
  2. I especially like the analysis of the code claims. The authors find that about 92% of claims about codebases fail (L243), which is surprising.
Critical weaknesses
  1. This paper is a scientometrics paper with some LLM prompting in it. Hence, the ML contribution of this paper is limited.
  2. The human evaluation in this work is limited. As far as I understand, only 60 papers were manually evaluated. On the contrary, the proposed "forensic deep read" is also done by LLMs.
Questions for authors
  1. How did you evaluate the GitHub links? How did you evaluate whether they were never public or were deleted?
  2. How should I interpret Table 3? What is the difference between "inappropriate performance metrics" and "Illegitimate features"? Did you use human evaluation for such a check?
1 comment
AuthorsOpenReview ↗

Clarifying Contribution to ML, Human Auditing, and Other Weaknesses

Hi! First, thanks for the time. I'll address each of the weaknesses and questions you posed in the same order to make sure this is equally clear.

W1. This paper is a scientometrics paper with some LLM prompting in it. Hence, the ML contribution of this paper is limited.

The way I'm understanding this is a scope objection, or rather that metascience papers don't necessarily count as viable contributions to ML. I think this is most strongly countered by ICLR's main-track call list, which welcomes "increasingly close collaboration and integration with scientific and humanistic fields outside the historical domain of ‘core ML’.”[7]”. The LLM is just a measurement instrument, while the contribution is what this study aims to establish about the validity and verifiability of ML research (and the limits of automated assessment itself), which are both possibly useful for guiding the field in new directions.

W2. The human evaluation in this work is limited. As far as I understand, only 60 papers were manually evaluated. On the contrary, the proposed "forensic deep read" is also done by LLMs.

Fair! Post the submission of this paper, we did also receive 3 more human-alignment spot checks, these including 100 samples each. All three human reviewers have different, but still biomedical-relevant backgrounds. We used these spot-checks specifically to ensure alignment on human notions of validity.

Q1. How did you evaluate the GitHub links? How did you evaluate whether they were never public or were deleted?

We looked first for (1) whether they were available at the time of our evaluation, or ever public, (2) did the repo actually contain contents beyond a README, and (3) if so, did the contents of the README allow for full replication of the methodology as expected.

For (1) - we checked both current access and historical evidence, primarily through GH Archive, with Software Heritage as a double check. The repository was classified as "no longer accessible" when qualifying evidence established both public availability and contents beyond a default README or empty repository. To be clear, this category includes both deletion and privatization, since Github uses the same status for both for public viewers. "Never public" in our case means no public availability could ever really be established.

Q2. How should I interpret Table 3? What is the difference between "inappropriate performance metrics" and "Illegitimate features"? Did you use human evaluation for such a check?

Table 3 is a breakdown of the occurrence rate for a selection of failure modes in papers whose methodology is primarily about ML-methodology in biomedical applications or lightly supported by ML in application. So, for instance, the table suggests that insufficient test set size is an incredibly common occurrence across both ML-primary and ML-supported papers, while illegitimate features are less-so.

Clarifying a bit on the definitions we give in Appendix C.1. Illegitimate features are information that the model wouldn't have received at prediction time. Example: predicting some diagnosis with a variable that would've been recorded only after that diagnosis. Inappropriate performance metrics are whether the reported measures actually support the primary scientific claim. Example: claiming reliable probability estimates while reporting only discrimination, and with no assessment of calibration.

We also performed human spot-checks for assessing the alignment and reliability of models used for this task (including building off of the data provided by previous work done at a smaller scale through predominantly human audit).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.