- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduced a pipeline to audit 27,200 biomedical papers published since 2011, with a particular focus on the 11,736 papers involving ML. The pipeline used LLMs to classify papers and screen for potential failure modes. The results were validated by rerunning the LLMs on a small subset of papers and conducting human reviews of a separate, smaller subset. The findings indicated that a strikingly high proportion of ML-related biomedical papers made claims that were not adequately supported.
Critical strengths
-
The paper introduces an LLM-assisted pipeline that classifies and audits biomedical papers.
-
The results are drawn from a large corpus (27,200 papers) that covers a long time span (2011-2016).
-
The (astonishing) findings of the ML-biomedical papers are supported by detailed statistical results.
Critical weaknesses
- The paper can be hard to understand for readers who are not very familiar with a few papers. The most important one is the leakage taxonomy of Kapoor & Narayanan [1]. Given that the paper currently lacks a Related Works section, it is highly recommended that the authors add a section discussing prior work in the field and their methodologies, and provide background.
[1] Kapoor, Sayash, and Arvind Narayanan. "Leakage and the reproducibility crisis in machine-learning-based science." Patterns 4.9 (2023).
- The manual validation process is not described in sufficient detail. Given the seriousness of the reported findings, which some readers may interpret as accusations, the authors should provide a more detailed account of the “accuracy control” procedure (Section 2.2) and the “Forensic Deep Read” process (Section 2.5). Please refer to the questions below for specific points requiring clarification.
Questions for authors
Dataset
- What is the overall quality of the 27,200 biomedical papers included in the corpus? The authors mention recording “the two-year mean citedness (OpenAlex’s open JIF analogue) and h-index” for each paper’s publication source, but is there any relationship between citation impact and the validity of the reported claims? For example, are more highly cited papers less likely to make inadequately supported claims? Are papers published in venues with higher h-indices more likely to provide reliable, well-maintained codebases?
Human Reviews
-
In line 97, the authors mentioned there is a blind human review of 60 papers, and the results are reported in the paragraph starting from line 328. However, since there is only one human rater, it is unclear if the mismatch between human and LLM classification is because of reasonable subjectivity or model defects. Comparing the statistics between human raters' alignment with the statistics between humans and LLMs can help to answer the question. Additionally, a sample with 60 points drawn from 27,200 samples is too small. Even if all 60 labels are correct, the one-sided 95% upper confidence bound on the true error rate can still be very low.
-
Section 2.5 introduced a forensic deep read on 200 papers. It would be helpful to include case studies for those papers when discussing the results. Additionally, why would the LLM detector fail in some cases?
-
For all the human reviews, what is the background of the annotators? How to make sure the quality of the reviews is reasonable?
Other
The abstract on OpenReview contains "\citep{kapoor2023leakage,lones2024avoid}".
3 comments
Please include the results from the spot-checks from the additional 3 human annotators in the camera-ready version if the paper is accepted.
I am slightly surprised by the efficiency of auditing 1000 papers. Can you share more details regarding the auditing process? For example, how many minutes do you spend on each paper? Is there a certain standard procedure?
I will keep my previous rating.
Copied from my clarification for other reviewer:
This audit was simply a human-eyes check for the following classifications:
- Does not render
- Non-english language (Largely a non-issue, since all the papers in languages I could understand were actually on-topic, though I still ruled them as failing this check).
- Irrelevant (not empirical ML/biomedical research - this included a very few off-topic papers (ML in fluids engineering without an explicit biomedical connection, eg) and a fair number of papers about adoption or reception of ML tools in the clinic.
- Conference proceedings/concatenations where our review pipeline might have conflated multiple texts
- No issue
I used a click-through interface which displayed the pdf full text of the paper, making it fairly easily to audit each text in between a few seconds (for instance, a paper in Korean or Russian was ruled out in <5 seconds at a glance) to 10-15 minutes (rarely, to assess whether a paper was actually biomedical - fluids engineering, for instance). The vast majority of the papers were either obviously relevant - a straightforward english publication about applying ML in bioscience, or about non-ML based biomedical research, without any concatenation - or obviously failing. Anything in a foreign language or concatenated or clearly about policy/adoption of ML was very quick to rule out. That's not to say I didn't have to click through papers and rest my eyes for the better part of a few days of course! These were not audits in the sense of the pipeline, that's my mistake for using the same word. This 1000-paper check was to ensure that the labeling and filtering applied properly and the pool of papers wasn't artifically inflated, which would drive down the apparent reproducibility.
Human Rater and Overall Quality
Hello! I've done some extra analysis.