acceptodds
Under review as a conference paper at ICLR 2027

PERSIST-Med: Evaluating Implicit Use of Patient History in LLM Memory Systems

Abstract

Patient-facing medical assistants must use earlier patient facts even when a new question does not mention them. A past drug reaction, for example, may still rule out a treatment today. Memory benchmarks largely test factual recall and updates, but rarely ask whether stored history changes a clinical decision unprompted. We introduce PERSIST-Med, a benchmark of 1,422 questions over 200 synthetic patient histories spanning 540 days. It distinguishes constraints that remain binding from values superseded by later evidence, and tests implicit application, update discrimination, conflict resolution, and abstention. Reference answers are specified from a guideline-grounded fact base before dialogues are generated. On a 40-patient evaluation set, two frontier models apply critical facts in 93.1–96.9% of implicit-application items with full history, versus 73.1–76.2% with mem0 or A-MEM, with about five times as many contraindicated recommendations. The systems fail at different stages: mem0 often fails to retrieve a fact it stored, while A-MEM often fails on items whose retrieved notes already contain the fact's text. On the full 200-patient corpus, adding a short, query-blind patient summary to mem0 reaches 94.1% on 797 implicit-application items, compared with 95.1% for full history. On 296 update items, however, its stale-value error rate remains 29.4%, compared with 1.0% for full history.

Primary area:
datasets and benchmarks
Keywords:
large language models, long-term memory, LLM memory systems, medical AI

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_JSPVOpenReview ↗
Rating
1
Confidence
3(−1)
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces PERSIST-Med, a fully synthetic healthcare question-answering benchmark designed to evaluate how large language model (LLM) memory systems use past patient history. The benchmark contains 1,422 questions across 200 synthetic patient histories, where each patient has 28 conversational sessions over a 540-day timeline. All 200 patients are generated by sampling from just 60 manually defined seed facts, which are grouped into long-term binding constraints (such as drug allergies or implants) and changing values (such as lab test results). The benchmark tests four tasks: implicit application of past constraints when the current question does not mention them, update discrimination between old and new lab values, conflict resolution when a patient claims an allergy is gone, and abstention when a fact is missing. Using LLM-as-a-judge (Gemini-3.1-Pro and Claude Sonnet 5) and a harm-weighted score, the authors compare putting the full patient history into the prompt (around 8.3K tokens per patient) against external memory systems like mem0, A-MEM, Letta, Zep, and naive RAG. The results show that frontier models with full context apply critical facts in 93% to 97% of implicit questions, while retrieval-based memory systems drop to 73% to 76% because they fail to retrieve unmentioned constraints or reason over retrieved notes. Finally, the authors show that pinning a short, query-blind patient summary into the prompt recovers most of the performance on implicit constraints, but still leaves high error rates on updated lab values.

Critical strengths
  • This paper uses medical scenarios to test how external memory systems perform compared to directly feeding the entire patient history into frontier models at once. The authors evaluate a good variety of recent agentic and retrieval memory systems (such as mem0/A-MEM).
Critical weaknesses
  • Lack of clinical validity and real-world grounding: Healthcare and medical benchmarks require rigorous evidence and validation. However, this entire 200-patient benchmark is generated from only 60 synthetic seed facts, and as the authors state in the limitations, none of the generated dialogues, clinical scenarios, or harm-tier weights were independently validated or supervised by physicians or clinicians. Generating synthetic conversations from 60 rule-based facts without expert annotation does not provide a trustworthy foundation for a medical benchmark. Even seeding the scenarios from real longitudinal EHR patient trajectories would be much more convincing than fully synthetic dialogues.

  • Limited technical contribution and generic problem framing: The main findings read more like a technical blog post or system evaluation report than a fundamental machine learning contribution for ICLR. Testing full-context prompting (which is only around 8.3K tokens per patient here) against top-k memory retrieval on unmentioned constraints is not unique to healthcare. This exact same retrieval failure mode applies to any domain where distant historical facts affect current answers without explicit keywords in the query, such as finance, supply chain, legal cases, or even personal assistants tracking dietary restrictions and personal tastes. Wrapping a generic retrieval versus long-context test in unverified synthetic medical dialogues does not make a strong contribution to either representation learning or clinical AI.

Questions for authors
  • Why did the authors not use real patient EHR trajectories as seeds, or involve physicians and clinical experts to supervise and validate the quality of the generated question-and-answer pairs and harm weights?

  • Since the core mechanism being tested (whether memory retrieval misses implicit long-term constraints compared to full-context prompting) is not unique to healthcare, why did the authors choose to frame this purely around unvalidated medical dialogues rather than general domains (such as personal dietary restrictions, finance, or legal histories) where synthetic setups are less clinically risky?

  • What is the core takeaway and future impact the authors hope PERSIST-Med will bring to the research community, doctors, or patients using LLMs? Are the practical insights from this benchmark offering anything fundamentally new beyond what is already captured in existing community guidelines and deployment practices (such as https://www.healthchatbotguide.org)?

Reviewer_VLyjOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses whether a medical assistant applies earlier patient information when the current query does not explicitly request it. PERSIST-Med contains 1,422 questions over 200 synthetic longitudinal histories and distinguishes persistent decision constraints from superseded values, alongside conflict-resolution and abstention tasks. Reference criteria are specified before dialogue generation. Experiments compare full context with several deployed memory systems and investigate storage, retrieval, and answer-stage failures. The authors find substantial implicit-application losses for retrieval-based memory with frontier answerers. A compact, query-blind pinned summary recovers much of this loss but does not resolve stale-value errors. Budget, renderer, judge, and model-strength controls qualify these conclusions.

Critical strengths
  1. The motivation identifies a meaningful mismatch between query relevance and decision relevance.

  2. The experiments go beyond aggregate benchmarking to test plausible explanations and repairs.

Critical weaknesses
  1. Reference and scoring validity remain insufficiently established. Clinical criteria and harm tiers lack independent clinician validation, and the authorship and possible LLM involvement in seed-level criteria are unclear. Cross-family judge agreement does not establish clinical correctness. Moreover, 51.4% of flagged stale-value answers contain both values; the lexical alternative does not resolve whether these reflect legitimate trajectory reasoning.

  2. The setting limits broader longitudinal conclusions. Histories are compact, the scale-up reuses 60 seed facts, and legitimate constraint release is untested. Since simple summaries already recover most T1 performance, a structured patient-profile baseline would clarify the challenge beyond maintaining standard safety-critical information. T4 dangerous-response rates and T5 abstention are also near saturation.

  3. A-MEM failure attribution is not fully established. Lexical presence of relevant terms does not demonstrate preservation of the complete actionable constraint. Some apparent answer-stage failures could therefore reflect lossy or ambiguous stored representations.

  4. Important qualifications are difficult to locate. Scoring ambiguities, attribution limitations, and judging regimes are dispersed across the appendices. Figure 2 combines configurations and evaluation subsets, making its comparisons unnecessarily difficult to follow.

Questions for authors
  1. How were the seed facts, trigger scenarios, and gold grading criteria authored and verified? Please clarify any LLM involvement separately from dialogue rendering and deterministic profile assembly. Could independent clinicians audit a stratified sample, particularly contraindication labels and T2 answers containing both values?

  2. How does summary pinning compare with a structured patient profile containing standard safety-critical fields? Would its benefits persist with greater constraint density, interacting facts, or unfamiliar fact types?

  3. Could matched histories include documented release or correction of a previously binding constraint? This would test whether pinning improves implicit application without creating inappropriate persistence.

  4. For A-MEM failures with lexical matches, could the retrieved context be audited for the complete actionable constraint? Replacing it with a verified formulation while holding the remaining context fixed would help distinguish representation loss from answer-stage non-application.

  5. Could the main text foreground the strict/permissive SDR distinction and attribution caveats, and report T4 correct/conservative rates alongside its saturated dangerous-response metric?

Reviewer_GrLzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a benchmark to test if LLMs can use patient history without queries directly referring to it. It compares different memory systems with full context, and concludes that the memory systems are significantly flawed for this use case.

Critical strengths

The paper goes in-depth on the different kinds of failure modes of memory systems, and the basic suggestion to include the whole context is valuable in itself.

Critical weaknesses
  1. Serious concerns about dataset construction and validation.

Although the authors claim to have reviewed all samples, the benchmark is low quality AI generated data, the release and use of it would harm the field more than help it. Specifically, inspecting the released examples reveals numerous obvious errors, including clinically inappropriate gold criteria, contradictory patient histories, and mechanically generated questions that do not fit the patient.

The section that describes dataset construction is also seemingly unconcerned with what models were used, and omits critical information regarding what human review was done, leaving the latter to a single line in the limitations section, whose extent and rigor are impossible to assess from the paper.

Examples include:

  • patient_022-Q04: a 34-year-old male persona asks about being 20 weeks pregnant, without any contextual explanation.
  • patient_025: a patient reports prior statin-induced rhabdomyolysis but is subsequently recorded taking atorvastatin, with the dose increasing from 20 to 40 mg. Different questions then require contradictory treatment of statins.
  • T4 uses the same de-labelling gold criterion for all 31 evaluation items, requiring allergist evaluation and supervised testing. This is inappropriate for conditions such as Stevens–Johnson syndrome and ACE-inhibitor angioedema, where the stated pathway cannot simply clear the precaution.
  • patient_083-Q05 in the larger release: a mechanical valve mentioned in the dialogue belongs to another person, but the gold answer treats it as the patient's own medical history.
  • patient_038: the persona changes relationship status and repeatedly changes the name of the same dog.

These are not subtle clinical disagreements. They suggest insufficient validation of both the generated conversations and the reference answers. I am particularly concerned by the discrepancy between these errors and the paper's statement that all items underwent author review.

  1. Limited effective benchmark size and questionable generalizability.

The benchmark contains 1,422 questions but draws on only 60 seed facts, with substantial repetition of questions and gold criteria. In the evaluation split, 34 of 36 abstention questions concern blood type, while T2 uses only six distinct query strings. These results therefore provide much less coverage than the headline item count suggests.

Furthermore, full patient histories contain only approximately 8K tokens. This is short enough for direct prompting, which limits conclusions about the settings where external memory systems are actually needed.

Questions for authors
  1. How exactly was the dataset created and validated? Which models generated the seed facts and dialogues? Where did the 60 seed facts originate? What did the claimed author review of every released item involve, and how did the examples above pass it?

  2. How were clinical gold criteria validated? In particular, why do all T4 items share the same supervised-testing criterion, including cases where such testing is inappropriate? Were reference answers reviewed independently of their generation?

  3. How representative are the results given the short histories and repeated queries? Can the authors evaluate longer histories or provide a stronger justification for the current setting?

Reviewer_6MhbOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PERSIST-Med, a benchmark for testing whether LLM memory systems use relevant patient history when the current question leaves it implicit. A short patient summary helps bridge much of the gap between mem0 and the full history on this task.

Critical strengths

The benchmark captures a practical problem in longitudinal care. A patient may ask about treatment without repeating an allergy or another detail that should affect the recommendation. Fixing the scoring criteria before generating the dialogues helps make this a controlled test.

The mem0 ablation is particularly informative. Giving the same answering model access to the entire memory store recovers 38 of 43 failures, providing direct evidence that many relevant facts were stored but missed by top-k retrieval.

Critical weaknesses

The clinical scoring rules lack independent clinician validation. For example, Appendix M lists first-generation cephalosporins as drugs to avoid without allergist clearance, although the AAAAI/ACAAI practice parameter distinguishes them by R1 side-chain similarity.

The dataset places lasting constraints early and current values late. This makes it hard to tell how much of the difference between the two tasks is due to when the relevant evidence appears.

The 200-patient evaluation reuses the same 60 seed facts and keeps histories at 27 sessions. It remains unclear whether the short summary would work as well with new clinical facts or longer histories.

Questions for authors

Could independent clinicians review a sample of the reference criteria and responses flagged for recommending a contraindicated option? What source supports the cephalosporin restriction in Appendix M?

Would summary pinning still help with applying lasting constraints more than with using current values if the relevant evidence were equally recent in both tasks?

How well does summary pinning work on held-out clinical facts or longer histories when the summary budget stays the same?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.