acceptodds
Under review as a conference paper at ICLR 2027

Are LLMs Up-to-date on Medical Knowledge?

Abstract

Biomedical evidence continuously evolves as scientific and clinical findings can be revised, nullified, or reversed. However, large language models (LLMs) still largely rely on static parametric memory acquired during training. Evaluating whether LLMs retain obsolete medical knowledge then becomes critical, yet confounded by a attribution challenge: standard benchmarks cannot separate knowledge staleness from a failure to acquire current medical knowledge during pretraining. To tackle this challenge, we introduce MEDLAPSE, an evaluation dataset derived from 1,787 author-corrected health-science finding lineages of up to 20 version updates across 14,172 multi-version medRxiv preprints. Specifically, each superseded or outdated finding is matched to two elements: its updated successor and paper-specific retention claims, which then helps verify whether a staleness might stem from outdated memory rather than complete unfamiliarity with the paper. Across 16 leading open-weight and commercial LLMs, they consistently select the superseded finding in 30.3% of queries in Q&A testing on average while the retained, unchanged claims are correctly recalled. Notably, this staleness rate remains largely invariant across parameter scale, medical domain pretraining, and training cutoff dates. Furthermore, applying existing, post-hoc model correction methods reveals critical limitations: while “forget-then-update” approach designed based on unlearning algorithms such as NPO and RMU achieve high efficacy or correction rate (>94%) using the training prompts, their performance on paraphrased ones drops up to 6.1% below the baseline. These results demonstrate that current model correction methods induce prompt-specific token suppression rather than genuine factual revision, highlighting that safely deprecating obsolete parametric medical knowledge in LLMs remains a non-trivial but critical research challenge.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
machine unlearning, large language models, factual staleness, evaluation benchmark, biomedical NLP

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_kMo1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a benchmark for evaluating whether LLMs retain outdated medical knowledge after scientific findings have been revised. The dataset contains medRxiv preprints, pairing outdated findings with updated ones to assess knowledge staleness. Experiments across 16 LLMs show that around 30% of responses still select outdated findings.

Critical strengths
  1. The paper builds benchmark datasets from real medical research revisions, with a useful design to distinguish outdated knowledge from unfamiliarity.
  2. The paper addresses medical knowledge staleness, an important issue for the reliability and safety of LLMs in healthcare.
Critical weaknesses
  1. Prior works like MedChangQA,TempoMed-Bench, and MedEditBench already shown LLMs give outdated answers, are inconsistent over time, and poor generalization after knowledge updates. MEDLAPSE introduces preprint revision histories and within-paper retention controls. However, the experiments don’t make clear what we learn from this design beyond confirming these known problems.
Questions for authors

What does MEDLAPSE benchmark reveal about LLM medical knowledge that existing benchmarks cannot?

2 comments
AuthorsOpenReview ↗

Response to Reviewer kMo1

What does MEDLAPSE benchmark reveal about LLM medical knowledge that existing benchmarks cannot?

Thank you very much for the question! We want to clarify the distinction between MEDLAPSE and existing works as follows.

Compared to existing works like MedChangeQA, TempoMed-Bench, and MedEditBench, in MEDLAPSE, we paid extra attention to the data curation pipeline with extra controls and properties such as within-paper topic controls, diverse probing formats, typed author revisions with clinical severity, cutoff-stratified primary-literature lineages, and also expert validation. Our dataset is much larger in size and revision linkages.

Furthermore, Medlapse is more meticulously curated than prior works because we want to ensure that (1) each revision is substantive (i.e., a real change in conclusion of an experiment) rather than just a typo fix or cosmetic revisions, (2) generated questions can be used for a closed-book exam to test the knowledge of an expert, not memory. For example, a claim can revise the number of data samples, but its conclusion still remains the same, so if we ask a question about the new data sample, it is only a memory test, which is not our purpose, and (3) same-paper retention control to ensure that a model is familiar with the topic of the claims and the retention claim is unique to each paper because we exclude the claims from other papers (via citation, references), by doing this, we ensure that extracted claims only reflect the paper's finding itself.

Table below summary our distinction compared with other datasets.

BenchmarkSourceRetention ControlFormatsRevision Types & SeverityCorrections Tested
MedChangeQACochrane reviews (512 pairs)No3-way verdictNoRetrieval
TempoMed-BenchClinical guidelines (721 MCQs)NoMCQ onlyNoRAG
MedEditBenchMissed static exam QANoMCQ / QANoFact editing
MEDLAPSE (ours)**Author preprint revisions (1,787 lineages; 29,666 items)Yes (same-paper control)MCQ, Open QA, RecognitionYes (3 revision types + 1–5 severity)Unlearning + RAG

Because of these differences, MEDLAPSE reveals three properties of LLM medical knowledge that existing benchmarks cannot isolate:

  1. Staleness (~30.3%) is not topical ignorance and scale, medical pretraining, and cutoffs do not fix it: By conditioning on papers where models correctly answer retention claims, all 16 LLMs still pick the superseded claim roughly 30.3%, regardless parameter scale, medical pretraining, cutoff. Thanks to such conditioning, our results avoid overclaims or inflation of the staleness.

  2. Which scientific updates fail and their clinical risk: We care also about the severity of the obsoleteness. For instance, epistemic nullifications cause ~50% of stale answers , while direction flips carry the highest clinical risk (2.73/5).

  3. Medical fine-tuning entrenches stale beliefs even against retrieval: Version-aware Hybrid RAG (Section 7) corrects 62.8% of stale answers on Llama-3 8B, but only 14.8% on OpenBioLLM-8B. This is quite surprising and it also reiterates (1), showing that having external access to the latest knowledge does not necessarily guarantee up-to-date responses, showing a nuanced challenge in examining the conflicting between internal and external knowledge in LLMs.

We will highlight these distinctions in §2 and §4-§7 in the revision.

AuthorsOpenReview ↗

Response to Reviewer kMo1

We apologize for submitting our rebuttal late with the discussion deadline near. We still hope you have an opportunity to review our clarification of what MEDLAPSE uniquely reveals about LLM medical knowledge, engage in any follow-up discussion, and consider updating your assessment if your concern is resolved.

Reviewer_JZruOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces MEDLAPSE, a benchmark built from 1,787 revised scientific findings across 1,194 medRxiv preprints, with unchanged within-paper claims used as retention controls. It generates ~30K questions in multiple formats and evaluates 16 LLMs. Among questions with correctly answered controls, models select the superseded finding in 30.3% of cases on average. The paper also evaluates unlearning, paraphrased probes, a clinical case study, and retrieval-based correction.

Critical strengths
  1. The benchmark uses actual scientific revisions and pairs superseded findings with updated findings and unchanged controls (Sec. 3.1). It covers different types of scientific changes and evaluation formats. This creates a useful resource for studying how models respond to evolving scientific claims.

  2. The evaluation covers a rigorous 16 LLMs and extends beyond multiple-choice accuracy to unlearning, paraphrased questions, clinical evidence synthesis, and RAG. The clinical case study finds a substantial difference between recognition and open-ended evidence responses. Very informative and useful in the conclusions.

Critical weaknesses
  1. The main attribution assumption is not fully justified. A correct response to an unchanged within-paper claim may reflect general domain knowledge or information contained in the question, rather than prior exposure to that paper. Selecting the superseded option does not necessarily demonstrate obsolete parametric memory. This limits how we interpret the staleness rate.

  2. The latest version of a preprint is not necessarily established medical ground truth. The benchmark primarily uses automated claim extraction and matching, and the main text does not establish independent clinical verification of all selected revisions. The paper also reports retracted papers within its collected sources. Additional expert validation would strengthen the interpretation of these revisions.

Questions for authors
  1. Can the authors validate whether retention-control accuracy reliably indicates exposure to the specific source paper, rather than general familiarity with the topic?

  2. What proportion of the revision lineages received independent clinical expert verification, and how are later-retracted papers or subsequently revised claims handled?

  3. Do the unlearning results on paraphrased probes differ from an update-only fine-tuning baseline, particularly for naturally retained rather than artificially introduced obsolete claims?

3 comments
AuthorsOpenReview ↗

Response to Reviewer JZru - 1 out of 2

W1.1, Q1. The main attribution assumption is not fully justified. A correct response to an unchanged within-paper claim may reflect general domain knowledge or information contained in the question, rather than prior exposure to that paper.

In a perfect experiment design, we want to condition on exposure to the original paper. However, it is impossible to do so using only model evaluation via metric such as accuracy, because such an evaluation might reflect all or a subset of (1) exposure or the original paper, (2) exposure to other papers citing the claims, (3) knowledge inference from relevant claims and facts, and potentially (4) random guess. As we can see, even if we have perfect access to the model’s training data with perfect lookup pipeline, there is no good way to isolate the contribution of (3) via model evaluation alone. Therefore, we totally agree with the reviewer that answering an unchanged control claim does not necessarily indicate prior exposure to the original paper.

As an effort to as much as possible to address this problem or to bring our model evaluation closer to match “prior paper exposure”, during the data curation process, we have made sure that the controlled retain claims are unique from other retain claims across all medrxiv papers (very low semantic similarity) that we collected, ensuring their specificity to a unique paper at least within our dataset. However, this is not perfect as you might already guess. It is then better to adopt a somewhat safer but still crucial claim: our conditioning conveys a connection with a broad sense of prior exposure of “familiar knowledge” rather than “original paper”. Therefore, we will fix a few writings in our paper as shown at the end of this response to make our paper’s contribution and assumption crystal clear to the readers. Overall, we think such conditioning on prior knowledge is still very important. Without such a condition as done in existing literature, we will greatly and unintentionally inflate or overestimate the knowledge staleness So, by conditioning on the control retain claims, we make sure that we only test on knowledge that the models have.

We will add this important discussion and assumptions and revise our manuscript as follows.

Abstract (Line 20): “rather than complete unfamiliarity with the paper” -> “rather than lack of relevant prior knowledge”

Introduction (Line 86-87): “it shows evidence that the model has previously seen the paper, so a superseded answer on the revised finding counts as memory rather than ignorance“ -> “it demonstrates relevant prior knowledge in the study context, helping distinguish superseded answers from topical unfamiliarity"

Introduction (Line 105): “separate outdated memory from an initial lack of exposure” → “separate outdated responses from a lack of relevant prior knowledge”

Section 3.1 (Line 156-157): “Correctly answering the control claim demonstrates that the model is assumed to be familiar with the paper and its topic” → “Correctly answering the control claim demonstrates that the model possesses relevant prior knowledge in the study’s context.”

W1.2 Selecting the superseded option does not necessarily demonstrate obsolete parametric memory. This limits how we interpret the staleness rate.

We partially agree with the reviewer that selecting the superseded does not necessarily demonstrate obsolete parametric memory, but only we provide one superseded option and three distractors. However, in our MCQ setting, each model will be given both superseded and latest findings for clearer discriminability, alongside two distractors (shuffle, 25% chance). As shown in Table 2, across 16 models, distractors are rarely chosen (11.5%), while the superseded and latest are 30.3% and 58.2%, respectively. Therefore, a stale response is not random guess, even when providing the latest option, models still pick superseded claims and it suggests two possible mechanisms: Models only have superseded knowledge during training and never acquired the updated findings. Dominant obsolete knowledge: both the superseded and latest knowledge are co-existed in the model, but the superseded claim dominates, causing it to outweigh the latest knowledge. For example, older findings had been widely cited by multiple works.

In either case, preferring superseded findings indicates that obsolete parametric memory controls the model’s prediction.

We will make this discussion more clear in our experiment design of the revision.

AuthorsOpenReview ↗

Response to Reviewer JZru - 2 out of 2

W2. The latest version of a preprint is not necessarily established medical ground truth. The benchmark primarily uses automated claim extraction and matching, and the main text does not establish independent clinical verification of all selected revisions. The paper also reports retracted papers within its collected sources. Additional expert validation would strengthen the interpretation of these revisions.

We agree that a revised preprint does not establish medical ground truth. We also noted in our Ethics Statement in Line 499: “A later preprint version is an author’s correction, not a peer-reviewed verdict” . However, this also does not mean that all the corrected scientific findings on pre-prints do not carry any scientific weight, especially when those are corrected by peer-reviewed, published articles. I think we can argue both ways, and either way does not discount the contribution of our work: scientific LLMs should at least take into account the latest findings, and we found that most of the time it defaults to the prior, outdated versions. To avoid conflating author revisions with clinical ground truth, we will revise our phrasing in §2 (Line 128-129) and §9 (Line 466-467) from "ground truth" to "author-corrected findings".

We will revise our manuscript to address each part of this weakness as follows:

Terminology: To avoid conflating author revisions with clinical ground truth, we will revise our phrasing in §2 (Line 128-129) and §9 (Line 466-467) from "ground truth" to "author-corrected findings".

Human verification: As detailed in Response to Q2, we verified 108 generated items (covering 88 revision lineages, 4.9%) with 11 annotators, achieving 87.0% acceptance for extraction and Q&A fidelity, alongside a biomedical graduate student’s review of the ICU case study. We will add this study to our revision.

Retracted papers: In Section 8 Limitation, we reported that 23 out of 14172 collected multi-version papers appear in the Retraction Watch, and we deliberately remove them before running our pipeline (claim extraction, lineage matching, and question generation). So these retracted papers are not in the final dataset. We will clarify information by explicitly stating those 23 papers have been excluded from the pipeline.

Q2. What proportion of the revision lineages received independent clinical expert verification, and how are later-retracted papers or subsequently revised claims handled?

We do not employ expert verification for MEDLAPSE, but we conducted an institutional IRB-approved human annotation with 11 college-level students in North America verifying 108 generated items (88 revision lineages and 20 retention claims). For each item, annotators inspected the original v1 and v_final claims (one version for a retention) alongside the generated question and answer. They judge whether the question and answer reflect the claims exactly and in case there are any mismatched revision lineages, annotators will reject that item. Annotators accepted 94 of the 108 items (87%).

However, for the case study, we employ a medical graduate student to verify the response from LLMs, covering 100 of 700 (14.2%) LLM responses in the experiments (all 10 evidence units). The expert agreed 71% with the judge from LLMs judges.

For later-retracted papers, we currently determined 23 retracted papers and excluded them from our dataset before running our pipeline. For subsequently revised claims, we crawl papers from Oct 2019 to June 2026, each lineage pairs the first version and the latest revision it appears.

We will add these statistics to our revision.

Q3. Do the unlearning results on paraphrased probes differ from an update-only fine-tuning baseline, particularly for naturally retained rather than artificially introduced obsolete claims?

We fine-tuned models on the latest version and tested on the same paraphrase questions with NPO, RMU. Two tables below show results for this experiment. Overall, fine tuning method also failed to generalize to paraphrases. Specifically, standard fine-tuning leaves naturally retained pre-cutoff claims flat (75.0%) compared to 75.7% before unlearning and gains on post-cutoff claims (35.1% vs 26.6%).

Pre-cutoff (natural claims)

Paraphrased-probe accuracy (%).

ModelBefore UnlearningFinetuningNPORMU
OLMo-372.872.891.069.6
Llama-3.178.480.197.972.5
MedGemma76.470.395.570.3
BioMistral75.376.892.870.7
Mean75.775.094.370.8

Post-cutoff (injected claims)

Paraphrased-probe accuracy (%).

ModelBefore UnlearningFinetuningNPORMU
OLMo-331.540.657.441.6
Llama-3.124.133.967.736.4
MedGemma27.635.746.538.3
BioMistral23.230.262.532.7
Mean26.635.158.537.3
AuthorsOpenReview ↗

Response to Reviewer JZru

We apologize for the response so close to the end of the author–reviewer discussion period. We are grateful for your supportive assessment and hope you still have a chance to review our new experiments and clarifications, and let us know if any points remain open.

Reviewer_TMRtOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MEDLAPSE, a benchmark for evaluating outdated medical knowledge in LLMs using revisions of scientific preprints. By pairing superseded findings with updated claims and unchanged control claims, the authors evaluate knowledge staleness across 16 LLMs and find that approximately 30% of responses select outdated findings. Experiments with unlearning methods further show limited generalization to unseen prompts, highlighting the difficulty of correcting obsolete knowledge.

Critical strengths
  1. The benchmark construction is interesting and addresses an important problem. Using scientific paper revisions to identify outdated findings provides a more realistic evaluation setting than synthetic knowledge updates.

  2. The experiments are fairly comprehensive, covering 16 models, multiple question formats, and several knowledge correction methods. The analysis of poor generalization after unlearning is particularly useful.

Critical weaknesses
  1. Correctly answering an unchanged control claim does not necessarily indicate prior exposure to the original paper, which weakens the attribution of outdated answers to memorized knowledge.

  2. Using the final preprint version as ground truth may be questionable, since author revisions do not always reflect established medical consensus.

  3. The correction experiments lack a standard fine-tuning baseline, making it difficult to determine whether the generalization failures are specific to unlearning methods.

Questions for authors
  1. How reliable is the unchanged-claim control as evidence of prior exposure? Have the authors considered validating this assumption against models with known training data?

  2. How are substantive scientific corrections distinguished from ordinary revisions or changes in interpretation? Is there any expert validation of the extracted revision lineages?

3 comments
AuthorsOpenReview ↗

Response to Reviewer TMRt - 1 out of 2

W1/Q1. How reliable is the unchanged-claim control as evidence of prior exposure? Have the authors considered validating this assumption against models with known training data?

In a perfect experiment design, we want to condition on exposure to the original paper. However, it is impossible to do so using only model evaluation via metric such as accuracy, because such an evaluation might reflect all or a subset of (1) exposure or the original paper, (2) exposure to other papers citing the claims, (3) knowledge inference from relevant claims and facts, and potentially (4) random guess. As we can see, even if we have perfect access to the model’s training data with perfect lookup pipeline, there is no good way to isolate the contribution of (3) via model evaluation alone. Therefore, we totally agree with the reviewer that answering an unchanged control claim does not necessarily indicate prior exposure to the original paper.

As an effort to as much as possible to address this problem or to bring our model evaluation closer to match “prior paper exposure”, during the data curation process, we have made sure that the controlled retain claims are unique from other retain claims across all medrxiv papers (very low semantic similarity) that we collected, ensuring their specificity to a unique paper at least within our dataset. However, this is not perfect as you might already guess. It is then better to adopt a somewhat safer but still crucial claim: our conditioning conveys a connection with a broad sense of prior exposure of “familiar knowledge” rather than “original paper”. Therefore, we will fix a few writings in our paper as shown at the end of this response to make our paper’s contribution and assumption crystal clear to the readers. Overall, we think such conditioning on prior knowledge is still very important. Without such a condition as done in existing literature, we will greatly and unintentionally inflate or overestimate the knowledge staleness So, by conditioning on the control retain claims, we make sure that we only test on knowledge that the models have.

We will add this important discussion and assumptions and revise our manuscript as follows.

Abstract (Line 20): “rather than complete unfamiliarity with the paper” -> “rather than lack of relevant prior knowledge”

Introduction (Line 86-87): “it shows evidence that the model has previously seen the paper, so a superseded answer on the revised finding counts as memory rather than ignorance“ -> “it demonstrates relevant prior knowledge in the study context, helping distinguish superseded answers from topical unfamiliarity"

Introduction (Line 105): “separate outdated memory from an initial lack of exposure” → “separate outdated responses from a lack of relevant prior knowledge”

Section 3.1 (Line 156-157): “Correctly answering the control claim demonstrates that the model is assumed to be familiar with the paper and its topic” → “Correctly answering the control claim demonstrates that the model possesses relevant prior knowledge in the study’s context.”

W2. Using the final preprint version as ground truth may be questionable, since author revisions do not always reflect established medical consensus.

We agree that a revised preprint does not establish medical ground truth. We also noted in our Ethics Statement in Line 499: “A later preprint version is an author’s correction, not a peer-reviewed verdict” . However, this does not mean that all the corrected scientific findings on pre-prints carry no scientific weight, especially when those are corrected by peer-reviewed, published articles. I think we can argue both ways, and either way does not discount the contribution of our work: scientific LLMs should at least take into account the latest findings, and we found that most of the time it defaults to the prior, outdated versions. To avoid conflating author revisions with clinical ground truth, we will revise our phrasing in §2 (Line 128-129) and §9 (Line 466-467) from "ground truth" to "author-corrected findings".

AuthorsOpenReview ↗

Response to Reviewer TMRt - 2 out of 2

W3. The correction experiments lack a standard fine-tuning baseline, making it difficult to determine whether the generalization failures are specific to unlearning methods.

We fine-tuned models on the latest version and tested on the same paraphrase questions with NPO, RMU. Two tables below show results for this experiment. Overall, both fine tuning and unlearning methods also failed to generalize to paraphrases, and hence generalization failures are not specific to unlearning methods. Specifically, standard fine-tuning leaves naturally retained pre-cutoff claims flat (75.0%) compared to 75.7% before unlearning and gains on post-cutoff claims (35.1% vs 26.6%).

Pre-cutoff (natural claims)

Paraphrased-probe accuracy (%).

ModelBefore UnlearningFinetuningNPORMU
OLMo-372.872.891.069.6
Llama-3.178.480.197.972.5
MedGemma76.470.395.570.3
BioMistral75.376.892.870.7
Mean75.775.094.370.8

Post-cutoff (injected claims)

Paraphrased-probe accuracy (%).

ModelBefore UnlearningFinetuningNPORMU
OLMo-331.540.657.441.6
Llama-3.124.133.967.736.4
MedGemma27.635.746.538.3
BioMistral23.230.262.532.7
Mean26.635.158.537.3

Q2.1 How are substantive scientific corrections distinguished from ordinary revisions or changes in interpretation?

As detailed in Appendix A.3, we leveraged the very capable Kimi-K2.5-Thinking model to assign revision type and substantiveness (True/False). A candidate revision is accepted only when the subject and object remain aligned and the change is substantive for closed-book evaluation. Furthermore, we employed three LLM judges (Qwen3, Deepseek, GPT-4.1-mini) and performed a major vote to filter out false positives. Below are two examples for rejected and accepted.

Accepted (nullification): v1: "We have shown that the median age and the average density of a country's population are statistically significantly associated with the average reproduction number." v2: "Our regression analysis has shown that the median age and the average density of a country's population are not statistically significantly associated with the average reproduction number." The two versions look almost identical, but one word, "not", turns a reported association between a country's demographics and COVID-19 transmission into a null finding.

Rejected (point-estimate change): v1: "The 226 primary cases were followed by 197 (49%) SARS-CoV-2 infections among 404 household contacts." v2: "The 226 primary cases were followed by 198 (49%) SARS-CoV-2 infections among 404 household contacts." One extra infection among 404 contacts leaves the attack rate at 49% and the conclusion unchanged. A question built on this pair could only test recall of a count, not knowledge of a finding.

Changes in interpretation are under the mechanism shifts, where empirical observation may remain unchanged while casual explanation changes (Example in Table 6). We keep this kind of revision because acting on an outdated mechanistic explanation might lead to different clinical or scientific inferences. We will add this clarification to the revised manuscript.

Q2.1. Is there expert validation of the extracted revision lineages?

Yes. To validate our question-generation pipeline, we conducted an institutional IRB-approved human annotation with 11 undergraduate students evaluating 108 generated items (88 revision lineages and 20 retention claims). For each item, annotators inspected the original v1 and v_final claims (one version for a retention) alongside the generated question and answer. They judge whether the question and answer reflect the claims exactly and in case there are any mismatched revision lineages, annotators will reject that item. Annotators accepted 94 of the 108 items (87%). We will include full details of this study in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer TMRt

We apologize for posting our rebuttal late with the author reviewer discussion deadline approaching. We hope the reviewer still has time to review our new validation experiments and engage in any remaining discussion and, if your concerns are addressed, please consider raising your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.