- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a benchmark for evaluating whether LLMs retain outdated medical knowledge after scientific findings have been revised. The dataset contains medRxiv preprints, pairing outdated findings with updated ones to assess knowledge staleness. Experiments across 16 LLMs show that around 30% of responses still select outdated findings.
Critical strengths
- The paper builds benchmark datasets from real medical research revisions, with a useful design to distinguish outdated knowledge from unfamiliarity.
- The paper addresses medical knowledge staleness, an important issue for the reliability and safety of LLMs in healthcare.
Critical weaknesses
- Prior works like MedChangQA,TempoMed-Bench, and MedEditBench already shown LLMs give outdated answers, are inconsistent over time, and poor generalization after knowledge updates. MEDLAPSE introduces preprint revision histories and within-paper retention controls. However, the experiments don’t make clear what we learn from this design beyond confirming these known problems.
Questions for authors
What does MEDLAPSE benchmark reveal about LLM medical knowledge that existing benchmarks cannot?
2 comments
Response to Reviewer kMo1
We apologize for submitting our rebuttal late with the discussion deadline near. We still hope you have an opportunity to review our clarification of what MEDLAPSE uniquely reveals about LLM medical knowledge, engage in any follow-up discussion, and consider updating your assessment if your concern is resolved.
Response to Reviewer kMo1
Thank you very much for the question! We want to clarify the distinction between MEDLAPSE and existing works as follows.
Compared to existing works like MedChangeQA, TempoMed-Bench, and MedEditBench, in MEDLAPSE, we paid extra attention to the data curation pipeline with extra controls and properties such as within-paper topic controls, diverse probing formats, typed author revisions with clinical severity, cutoff-stratified primary-literature lineages, and also expert validation. Our dataset is much larger in size and revision linkages.
Furthermore, Medlapse is more meticulously curated than prior works because we want to ensure that (1) each revision is substantive (i.e., a real change in conclusion of an experiment) rather than just a typo fix or cosmetic revisions, (2) generated questions can be used for a closed-book exam to test the knowledge of an expert, not memory. For example, a claim can revise the number of data samples, but its conclusion still remains the same, so if we ask a question about the new data sample, it is only a memory test, which is not our purpose, and (3) same-paper retention control to ensure that a model is familiar with the topic of the claims and the retention claim is unique to each paper because we exclude the claims from other papers (via citation, references), by doing this, we ensure that extracted claims only reflect the paper's finding itself.
Table below summary our distinction compared with other datasets.
Because of these differences, MEDLAPSE reveals three properties of LLM medical knowledge that existing benchmarks cannot isolate:
Staleness (~30.3%) is not topical ignorance and scale, medical pretraining, and cutoffs do not fix it: By conditioning on papers where models correctly answer retention claims, all 16 LLMs still pick the superseded claim roughly 30.3%, regardless parameter scale, medical pretraining, cutoff. Thanks to such conditioning, our results avoid overclaims or inflation of the staleness.
Which scientific updates fail and their clinical risk: We care also about the severity of the obsoleteness. For instance, epistemic nullifications cause ~50% of stale answers , while direction flips carry the highest clinical risk (2.73/5).
Medical fine-tuning entrenches stale beliefs even against retrieval: Version-aware Hybrid RAG (Section 7) corrects 62.8% of stale answers on Llama-3 8B, but only 14.8% on OpenBioLLM-8B. This is quite surprising and it also reiterates (1), showing that having external access to the latest knowledge does not necessarily guarantee up-to-date responses, showing a nuanced challenge in examining the conflicting between internal and external knowledge in LLMs.
We will highlight these distinctions in §2 and §4-§7 in the revision.