- Rating
- 1
- Confidence
- 3(−1)
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces PERSIST-Med, a fully synthetic healthcare question-answering benchmark designed to evaluate how large language model (LLM) memory systems use past patient history. The benchmark contains 1,422 questions across 200 synthetic patient histories, where each patient has 28 conversational sessions over a 540-day timeline. All 200 patients are generated by sampling from just 60 manually defined seed facts, which are grouped into long-term binding constraints (such as drug allergies or implants) and changing values (such as lab test results). The benchmark tests four tasks: implicit application of past constraints when the current question does not mention them, update discrimination between old and new lab values, conflict resolution when a patient claims an allergy is gone, and abstention when a fact is missing. Using LLM-as-a-judge (Gemini-3.1-Pro and Claude Sonnet 5) and a harm-weighted score, the authors compare putting the full patient history into the prompt (around 8.3K tokens per patient) against external memory systems like mem0, A-MEM, Letta, Zep, and naive RAG. The results show that frontier models with full context apply critical facts in 93% to 97% of implicit questions, while retrieval-based memory systems drop to 73% to 76% because they fail to retrieve unmentioned constraints or reason over retrieved notes. Finally, the authors show that pinning a short, query-blind patient summary into the prompt recovers most of the performance on implicit constraints, but still leaves high error rates on updated lab values.
Critical strengths
- This paper uses medical scenarios to test how external memory systems perform compared to directly feeding the entire patient history into frontier models at once. The authors evaluate a good variety of recent agentic and retrieval memory systems (such as mem0/A-MEM).
Critical weaknesses
-
Lack of clinical validity and real-world grounding: Healthcare and medical benchmarks require rigorous evidence and validation. However, this entire 200-patient benchmark is generated from only 60 synthetic seed facts, and as the authors state in the limitations, none of the generated dialogues, clinical scenarios, or harm-tier weights were independently validated or supervised by physicians or clinicians. Generating synthetic conversations from 60 rule-based facts without expert annotation does not provide a trustworthy foundation for a medical benchmark. Even seeding the scenarios from real longitudinal EHR patient trajectories would be much more convincing than fully synthetic dialogues.
-
Limited technical contribution and generic problem framing: The main findings read more like a technical blog post or system evaluation report than a fundamental machine learning contribution for ICLR. Testing full-context prompting (which is only around 8.3K tokens per patient here) against top-k memory retrieval on unmentioned constraints is not unique to healthcare. This exact same retrieval failure mode applies to any domain where distant historical facts affect current answers without explicit keywords in the query, such as finance, supply chain, legal cases, or even personal assistants tracking dietary restrictions and personal tastes. Wrapping a generic retrieval versus long-context test in unverified synthetic medical dialogues does not make a strong contribution to either representation learning or clinical AI.
Questions for authors
-
Why did the authors not use real patient EHR trajectories as seeds, or involve physicians and clinical experts to supervise and validate the quality of the generated question-and-answer pairs and harm weights?
-
Since the core mechanism being tested (whether memory retrieval misses implicit long-term constraints compared to full-context prompting) is not unique to healthcare, why did the authors choose to frame this purely around unvalidated medical dialogues rather than general domains (such as personal dietary restrictions, finance, or legal histories) where synthetic setups are less clinically risky?
-
What is the core takeaway and future impact the authors hope PERSIST-Med will bring to the research community, doctors, or patients using LLMs? Are the practical insights from this benchmark offering anything fundamentally new beyond what is already captured in existing community guidelines and deployment practices (such as https://www.healthchatbotguide.org)?