acceptodds
Under review as a conference paper at ICLR 2027

Evaluating and Understanding Agent Memory Systems for Chronic Care

27 views

Abstract

Memory is essential for large language model (LLM) agents in chronic care, where patient histories span years of visits across multiple departments. Many memory systems have been proposed for LLM agents, yet existing benchmarks, drawn from general-domain dialogues or medical records over limited years of visits and departments, cannot reveal how they behave or why they fail in chronic care. To fill this gap, we introduce MEME-Bench, a clinically grounded benchmark of patient histories synthesized from real chronic-care records, with records spanning over a decade, patients up to 1,718 visits, and 1,646 questions on eight capabilities asked as the history accumulates. Leveraging MEME-Bench, we evaluate 11 memory systems and a long-context baseline across seven backbones and uncover three critical findings: (1) no memory system dominates across capabilities, and none outperforms the long-context baseline on every axis; (2) longitudinal trajectory tracking and cross-department integration remain poor for all systems; and (3) tracing the required facts through the memory pipeline reveals retrieval as the most common source of failure, rooted in evidence scattered across departments, recurring measurements and precise medical terminology. Controlled ablations further show that combining agentic retrieval with date-filtered search improves both of the hardest axes, whereas prompting cannot replace these components.

Primary area:
datasets and benchmarks
Keywords:
Chronic Care, Agent Memory Systems

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bJaROpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on evaluating the long-term memory capabilities of LLM agents in chronic care scenarios. The authors point out that existing general-domain or short-term medical memory benchmarks fail to reflect the complex challenges of real-world chronic disease management, which involve long time horizons, numerous departments, and massive follow-up records. To address this, the paper proposes a new benchmark, MEME-Bench, synthesized from real chronic-care records.

Critical strengths
  1. The scale of MEME-Bench (often thousands of visits and 90k+ tokens) and its 8-dimensional design are highly systematic.
  2. The paper abstracts memory systems into a four-stage pipeline, pinpointing "retrieval failure" as the core bottleneck by tracking the flow of each fact through the pipeline.
  3. The study covers 11 current mainstream memory architectures and 7 different backbones.
Critical weaknesses
  1. The core experimental data is synthesized from real chronic-care records. However, synthetic data often carries generation artifacts, overly smooth logic, or structural biases, which may cause the failure modes of memory systems (such as certain retrieval failures) to deviate from the noise, typos, and unstructured doctor notes found in real-world electronic health records.

  2. With the proliferation of ultra-long context models (supporting millions of tokens), the capacity of direct long-context attention is rising rapidly. Although the paper includes a long-context baseline, it lacks a fine-grained "context length vs. performance degradation curve" to reveal the exact threshold at which memory systems truly outperform long-context models.

  3. Although the paper emphasizes that MEME-Bench contains dynamic time-point questions, these questions and evaluation suites remain essentially offline-preset and statically evaluated. In real clinical interactions, patient disease evolution is highly open-ended and adversarial.

Questions for authors

See above.

6 comments

We thank the reviewer for the detailed and constructive review. We answer W1–W3 in the order of the review: (All additional results added in blue color text in appendix)

W1 “...synthetic data …may cause the failure modes of memory systems (such as certain retrieval failures) to deviate from the noise, typos, and unstructured doctor notes found in real-world electronic health records.”

We add noises (e.g. random non-clinical sentences, random change of full phrases into abbreviations, copy-over prior conversations, change date formats, etc.) and typos to every visit, affecting approximately 33% of the turns, 1.6% of the total token count, and see if our conclusions still hold on Qwen 3.6 27B.

Our main conclusions remain consistent. MIRIX, MedCache, Letta, REMem remain the first tier of memory systems.

Original hardest axes, Trajectory and Cross-department remain the hardest challenges.

What changed is that stronger agentic systems dropped more with noise addition, as we can observe Letta/MIRIX/REMem average loses .09 while the others suffer less.

Performance comparison of 11 memory systems and long context across 8 axes with and without added noise.

methodsConditionRecallStateSafetyEntityAbst.Traj.X-dept.Valid.Mean
Long contextNormal0.760.790.610.880.660.250.100.430.56
Noise0.620.590.580.770.640.160.060.350.47
BM25 RAGNormal0.640.440.700.830.420.190.120.450.47
Noise0.500.400.500.750.380.160.130.330.39
Dense RAGNormal0.610.610.700.850.540.220.130.370.50
Noise0.570.470.650.790.490.200.160.440.47
HippoRAG 2Normal0.670.520.800.830.420.340.180.600.55
Noise0.620.470.720.800.430.310.160.530.51
A-MemNormal0.620.610.740.830.540.270.190.350.52
Noise0.580.470.700.670.480.160.110.320.44
MemOSNormal0.670.480.830.810.450.260.170.500.52
Noise0.620.440.680.720.420.320.200.420.48
Mem0Normal0.540.510.720.830.380.240.050.380.46
Noise0.480.470.580.730.370.210.070.330.41
LightMemNormal0.700.510.690.820.410.260.110.310.48
Noise0.630.420.670.800.450.250.040.240.44
LettaNormal0.740.790.890.730.820.470.450.750.71
Noise0.620.650.750.670.770.370.440.630.61
MIRIXNormal0.800.760.900.680.890.520.470.700.72
Noise0.780.650.750.530.820.350.420.660.62
REMemNormal0.730.550.830.690.920.580.190.760.66
Noise0.660.410.670.800.790.370.130.750.57
MedCacheNormal0.690.680.740.680.990.320.510.430.63
Noise0.670.660.760.710.960.300.480.380.62

W2 “...lacks a fine-grained "context length vs. performance degradation curve...”

We did not include an isolated context length vs performance degradation because it entangles the number of evidence visits required, the context length accumulated when it is asked, and its evidence depth, defined as the token distance to the supporting evidence.

Fig 9-19 in appendix already visualizes this effect of these three axes on memory system performance, and we break it down below into tables. From this analysis, we observe that:

Evidence depth is the only axis where Long Ctx baseline shows wins. It is the best method from 32K to 128K, with the evidence inside its window. Beyond the window, it drops to 0.40, while the memory systems stay roughly flat.

On record length, Long Ctx never leads. The < 128K bin, where the whole record fits, is mostly Temporal Validity Inference questions, which is where Long Ctx is weak.

On number of evidence visits required, the best memory system leads Long Ctx at every count. The gap is significant at 2 to 9 visits. Every method dips at 2 visits.

Performance of memory systems and long context under different evidence and record conditions.

Number of evidence visits required v.s. performance

Method123–45–9≥ 10
Long Ctx0.640.320.440.490.61
BM250.650.330.400.370.42
Dense0.620.300.380.440.47
Letta0.790.560.540.590.69
MIRIX0.710.570.580.640.67
ReMem0.730.420.580.610.57
HippoRAG0.740.350.500.470.52
A-Mem0.610.340.380.480.47
Mem00.650.220.310.440.47
LightMem0.580.310.380.440.49

Record length v.s. performance

Method< 128K128–256K256–384K≥ 384K
Long Ctx0.400.450.590.39
BM250.500.370.520.36
Dense0.460.360.520.41
Letta0.850.620.670.49
MIRIX0.700.650.640.60
ReMem0.840.610.590.55
HippoRAG0.710.490.570.46
A-Mem0.400.390.530.42
Mem00.400.340.500.41
LightMem0.340.330.530.41

Evidence depth v.s. performance

Method< 32K32–64K64–128K≥ 128K
Long Ctx0.580.670.660.40
BM250.540.510.340.43
Dense0.460.470.490.46
Letta0.750.620.580.60
MIRIX0.730.620.600.59
ReMem0.730.480.500.58
HippoRAG0.640.510.450.52
A-Mem0.470.510.490.47
Mem00.480.430.450.43
LightMem0.460.450.450.45

W3 “Although the paper emphasizes that MEME-Bench contains dynamic time-point questions, these questions and evaluation suites remain essentially offline-preset and statically evaluated. In real clinical interactions, patient disease evolution is highly open-ended and adversarial.”

MEME-Bench is evaluated as a stream. Each system ingests visits one at a time in chronological order, and questions are posed at intermediate checkpoints using only the history up to that point, so memory must be maintained while the record grows (Section 3). The questions mirror decisions made during chronic follow-up: whether a referral was completed and what it found, whether a temporary restriction still applies, which drug is safe given recorded constraints, and how a measurement has changed. Closed-loop evaluation, in which an agent's decisions change the patient's later course, needs a patient simulator whose dynamics respond to treatment. We agree it is the natural next step and will add it to the conclusion as future work. MEME-Bench opens the door for future works that involve such evolving clinical world models or realistic longitudinal patient digital twins.

Reviewer_eYzcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This article introduces MEME-Bench, a benchmark for evaluating agent memory systems in long-term chronic disease care scenarios. The benchmark includes 23 synthetic patient histories built from real clinical records. The authors evaluated 11 memory systems and a long contextual baseline model on seven base models. The results indicate that no single system consistently outperforms others across all capability dimensions. The authors found that information retrieval is the main bottleneck and highlighted the advantages of agency retrieval and date-filtered search through gold-fact tracing, oracle evidence interventions, and component ablations.

Critical strengths
  1. This article introduces a large-scale benchmark test that covers patient histories among a long time and involving multiple departments, as well as eight clinically significant memory abilities. It can capture a variety of complex scenarios, including the dynamic evolution of patient status, historical measurement data, and the integration of cross-departmental evidence, thus providing a valuable testing platform for the memory system of intelligent agents.
  2. This paper evaluates 11 memory systems under seven backbone models and provides extensive comparisons with existing methods. It also analyzes the failure modes of different memory architectures by tracking the gold facts in the memory pipeline, implementing oracle interventions, and conducting component ablation experiments.
Critical weaknesses
  1. The validation of benchmark ground-truth is insufficient. This benchmark relies on multiple stages of data generation based on LLM. The questions and key evidence come from structured events and chains of evidence, while the memory system evaluates based on the generated doctor-patient dialogue. Although the authors reported expert review and human verification, it remains unclear whether all the necessary golden facts were preserved in the final conversation. This is particularly important because missing or inconsistent evidence not only affects the accuracy of the baseline but also subsequent failure attribution analysis.
  2. The experimental controls for failure attribution and design conclusions is insufficient. The oracle interventions do not fully isolate retrieval failures from memory construction failures, as Oracle Retrieval bypasses both stages while Oracle Memory only addresses storage. Furthermore, the agentic retrieval ablation does not clearly control for retrieval budget, making it difficult to distinguish the benefits of agentic planning from additional retrieval opportunities. The date-filtering experiments are also primarily conducted on Letta, limiting the generalizability of the proposed design conclusions across memory architectures.
Questions for authors
  1. How do the authors verify that the gold fact are faithfully preserved in the final generated doctor-patient dialogues? Could the authors provide quantitative statistics on the coverage of these facts, or elaborate on the manual verification process?
  2. Since Oracle Retrieval bypasses potential failures in memory construction and retrieval, while Oracle Memory addresses only the storage aspect, how do the authors disentangle the impacts of these distinct failure sources? Could the authors provide a more rigorous comparative analysis to substantiate the claim that "retrieval is the primary bottleneck"?
  3. In the ablation studies concerning agentic retrieval, was this approach compared against a single-shot retrieval variant using an equivalent retrieval budget (in terms of the number of retrieved tokens or computational cost)? If not, to what extent can the observed performance gains be attributed to agentic retrieval itself, rather than simply to the additional retrieval opportunities provided?
  4. The ablation experiments regarding date-based filtering were conducted primarily within the Letta framework. Can similar performance improvements be expected for other memory architectures, particularly those that overwrite historical facts or rely on graph-based retrieval? Furthermore, could the authors explain why date-based filtering yields such significant performance gains in cross-departmental integration tasks?
6 comments

We thank the reviewer for the detailed and constructive review. We answer W1–W4 in the order of the review: (All additional results added in blue color text in appendix)

“How do the authors verify that the gold fact are faithfully preserved in the final generated doctor-patient dialogues? Could the authors provide quantitative statistics on the coverage of these facts, or elaborate on the manual verification process?”

Fidelity is controlled at generation time and audited afterwards. We show quantitative comparison of evidence and dialogue comparison below:

Each event-chain fact is an atom with an ID. The doctor agent receives, turn by turn, the atom IDs it must convey, and each generated turn reports the IDs it actually conveys. Grounding rules forbid inventing or altering values, doses, dates and diagnoses. Small talks to add conversational realism are limited to non-clinical topics.

We used claude opus 5 to compare every event-chain fact with the dialogue of its visit and labeled it correctly expressed, omitted, or inconsistent (changed value or missing detail). Additionally, during benchmark construction, each question was answered correctly by GPT-5.6-Sol given its gold evidence visits, so a question whose required fact was lost would have been removed.

Statistics of how many facts are omitted or changed from the source event chain.

Fact fieldFactsOmittedChanged
Diagnosis30,5570.3%0.7%
Patient's own description99,8022.3%1.6%
Tests and medication98,4862.6%3.7%
Doctor's notes72,8862.1%1.0%
Doctor's plans30,5232.1%0.6%
Health advice4,2203.9%0.6%
Doctor's extra notes22,5121.9%3.2%
Vital signs4,1160.4%0.0%
All facts363,1022.1%2.0%

Clinician assessment. Clinicians rated 100 randomly sampled visit dialogues on a 5-point scale (100 ratings per question), showing the high fidelity of generated dialogues.

Rating assessment of 100 randomly sampled dialogues by clinicians.

QuestionRatings12345Mean ± SDRated 4 or 5
Q1. Fits the condition1000.0%0.0%13.0%59.0%28.0%4.15 ± 0.6387.0%
Q2. Grounded in what appeared before1001.0%0.0%3.0%23.0%73.0%4.67 ± 0.6496.0%
Q3. Doctor-Patient Role Play1000.0%0.0%6.0%60.0%34.0%4.28 ± 0.5794.0%
Q4. Alignment with real clinical interactions1000.0%4.0%17.0%54.0%25.0%4.00 ± 0.7779.0%

“Since Oracle Retrieval bypasses potential failures in memory construction and retrieval, while Oracle Memory addresses only the storage aspect, how do the authors disentangle the impacts of these distinct failure sources? Could the authors provide a more rigorous comparative analysis to substantiate the claim that "retrieval is the primary bottleneck"?”

Both Oracle Retrieval and Oracle Memory inject the same information: for each required fact group, the verbatim dialogue turn that states it, with its visit header. Oracle memory (OM) restores the memory each system built in its original run, checkpoint by checkpoint, and writes the payload into the system's own store in that store's record format, skipping any LLM rewriting; indexing, retrieval and answering are unchanged. Oracle retrieval (OR) adds the same payload to the reader's context, next to what the system retrieved itself. Therefore ΔOR − ΔOM is the loss that remains when the evidence is stored but must still be retrieved. Additionally, Figure 6 displays in detail Attribution of errors by gold facts retained at different stages of the memory lifecycle for representative methods.

MethodDelta OM (storage fixed)Delta OR (storage + retrieval fixed)Delta OR - Delta OM (retrieval loss with perfect storage)
Letta+11.3 [+7.7, +14.5]+18.5 [+14.8, +22.1]+7.2 [+4.4, +10.1]
HippoRAG+8.6 [+5.1, +12.7]+27.9 [+23.5, +32.8]+19.2 [+15.5, +22.6]
ReMem+1.2 [-2.2, +5.4]+25.3 [+20.9, +30.4]+24.1 [+20.2, +27.6]
A-Mem+16.1 [+12.6, +19.8]+27.9 [+24.9, +31.8]+11.7 [+8.5, +15.7]
Dense+16.3 [+12.8, +20.6]+29.4 [+25.8, +34.5]+13.0 [+10.3, +16.5]
MemOS+3.7 [+0.3, +7.1]+42.7 [+38.6, +46.2]+39.0 [+36.0, +42.0]
BM25-0.4 [-2.5, +1.5]+37.1 [+34.6, +39.4]+37.5 [+35.2, +39.9]
LightMem+8.4 [+4.5, +12.3]+28.5 [+24.8, +32.9]+20.0 [+16.9, +23.5]
MIRIX-1.2 [-3.1, +0.7]+9.1 [+6.6, +11.5]+10.4 [+7.7, +13.0]
Mem0+6.5 [+4.4, +9.2]+33.6 [+30.8, +36.2]+27.0 [+23.6, +30.3]

“In the ablation studies concerning agentic retrieval, was this approach compared against a single-shot retrieval variant using an equivalent retrieval budget (in terms of the number of retrieved tokens or computational cost)? If not, to what extent can the observed performance gains be attributed to agentic retrieval itself, rather than simply to the additional retrieval opportunities provided?”

Single shot. We want to note that without agentic systems, it won’t be possible to get additional rounds of results because the query will be the same. Agentic systems reformulate the query based on previous round returns.

Why agentic retrieval helps.

  1. Query reformulation. Records use abbreviations that a question may spell out. Example (trajectory question, patient P18): "Between 2010-05-03 and 2011-10-24, how did this patient's international normalized ratio change over the readings taken in that period?" The record writes only "INR", in five visits. BM25 gives zero weight to "international" and "normalized", which never occur; its top five visits match generic words ("period", "readings", "taken", "ratio"), and the first evidence visit ranks 22nd, matched only through its date.
  2. Iterative search. An agent searches, reads the results and searches again. Consistent with this, agentic systems gain the most from backbone scaling (Fig. 4).
  3. Self-check search results. An agent decides whether the retrieved information is relevant and changes its query accordingly.

A.12 Top-K ablation studies effect of retrieval budget on overall performance. We vary the number of top-K items returned by each memory method (Figure 20 and 21). Most methods improve substantially as K grows from 3 to 20. Every method achieves higher accuracy at K = 20 than at K = 3.(Figure 20) This suggests that part of the evidence missed under a small K retrieval can be retrieved by using a larger retrieval budget. Letta and MIRIX are near a plateau. The benefit of increasing K varies across methods. Letta and MIRIX are near a plateau with a larger K and show the smallest overall gains from K=3 to K=20, and ReMem with agentic loop also gains little beyond K = 10, whereas several non-agentic retrieval methods are still improving from K=10 to 20. (Figure 21). Gains differ across axes, and clinical state updating gains the most.For this axis, we design questions on measuring whether the memory methods can track the current state of clinical facts. In chronic care, one clinical fact looks much like the many earlier follow-ups, and the similarity ranking ignores dates, so these visits can fall anywhere in the retrieval ranking, and a larger K can include them.

Finally, we provide additional experiments that show more precise control of the round budget. For R=1, we can see the top agentic methods still perform the best (Letta MiRIX). For non-agentic methods, we adapt the ability to reformulate query from Letta and perform more rounds of search, and observe great performance improvements, showcasing the importance of agentic retrieval.

Performance of 11 memory systems under matched retrieval budget measured in retrieval rounds.

MethodR = 1R = 3R = 5R = 10
Letta0.6090.6650.6760.689
MIRIX0.6060.6720.6830.692
MedCache0.4950.5680.5850.608
Mem00.4780.5410.5270.571
HippoRAG0.550.6180.6420.681
ReMem0.5040.5750.5980.645
A-Mem0.520.6420.6410.665
Dense RAG0.500.6050.6550.631
MemOS0.520.6280.6480.648
BM25 RAG0.470.6280.6710.671
LightMem0.490.4910.5140.554
Reviewer_rQRPOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce MEME-Bench, a benchmark for evaluating LLM agent memory systems in chronic care. The benchmark consists of synthetic patient histories generated from real clinical records, with 1,646 questions across eight capabilities. The authors evaluate 11 memory systems across seven LLM backbones, and further analyze where memory systems fail across different stages. They find that retrieval is a major bottleneck, and that agentic retrieval combined with date filtering improves performance.

Critical strengths
  • I think evaluating memory systems over long clinical histories is an important problem, especially since most existing medical benchmarks have much shorter histories.

  • The eight evaluation axes seem reasonable, especially longitudinal trajectory tracking and cross-department integration, which are important for chronic care.

  • There is a good amount of experimentation across memory systems and different backbones. I also appreciate the oracle evidence experiments and the ablations to understand where the systems fail.

Critical weaknesses
  • I am quite concerned by the enormous amount of LLM usage in constructing the benchmark. From Section 3 and Appendix B, the original clinical records are abstracted, after which LLMs regenerate clinical events, measurements, treatments, and doctor-patient dialogues. In Appendix B.3, the authors even mention adding entirely fictional drug allergies and family histories. While I understand the privacy motivation, I am not convinced these generated histories preserve the actual difficulties of real clinical records. I would expect some quantitative comparison between the original and synthetic data distributions, especially given the claims about clinical realism.

  • The authors mention clinician verification and correcting or modifying 639 question-answer pairs, which is good. However, the questions are generated from the evidence chains, whereas the memory systems operate on LLM generated dialogues. How often are clinical facts omitted or incorrectly represented during dialogue generation? Seeing the extensive prompts in Appendix A.9, I also feel the generated conversations are quite constrained. I would expect some quantitative analysis of dialogue fidelity and how representative these conversations are of real clinical interactions.

  • The authors use GPT-5.6-Sol to filter questions and then as the evaluation judge. Although human agreement is reported, Table 5 shows substantial differences for some axes. For Cross-Department Integration, GPT has a pass rate of 0.24 compared with 0.42 for both clinicians. Moreover, questions that GPT-5.6 cannot answer even with gold evidence are filtered out (Section 3, L245-252). I feel this may introduce a selection bias toward questions that this particular model can solve. How many questions were removed this way, and would the conclusions remain similar with human evaluation of those questions? Do the questions that get removed actually not good?

  • There is a bit of a lack of stronger retrieval baselines. Although the authors compare against BM25 and Dense RAG, a major conclusion is that agentic retrieval with date filtering performs better. Given that many questions explicitly involve dates, measurements, and departments, I would expect standard RAG with temporal or metadata filtering to already work quite well. Why not compare against BM25 or Dense RAG with date filtering, or a simple hybrid retrieval method?

Overall I think the paper tries to solve a good problem and has a decent amount of experimentation. However, I am not convinced by the validity of the heavily LLM-generated benchmark, especially given how much of the original clinical information is regenerated. There are also open questions about the evaluation protocol and experimental confounders in the retrieval comparisons. I feel some of the conclusions are stronger than what the experiments support. I lean to reject in the current state.

Questions for authors

Mentioned along with weaknesses.

8 comments

We thank the reviewer for the detailed and constructive review. We answer W1–W4 in the order of the review: (All additional results added in blue color text in appendix)

W1. Clinical realism and the extent of synthetic generation

MEME-Bench keeps the longitudinal structure of 23 real chronic-care histories and regenerates their content. Kept from the source records: the visit sequence, departments, relative intervals, disease trajectories (as trends and states relative to clinical targets) and treatment-adjustment patterns. Regenerated: calendar dates, measurement values, regimens, narrative text and dialogue, generated under public clinical guidelines and checked by three clinicians (Appendices A.4, A.16). This design is what allows the benchmark to be released publicly contributing to the research community, while the source records stay private.

Injected safety constraints and family histories. We inject synthetic items because the Safety Constraints and Entity Attribution axes need drug constraints and the histories of people other than the patient (e.g., family members), which source records document too sparsely to test reliably. MEME-Bench measures the memory of LLM systems rather than their medical knowledge, so an injected item is valid if it is clinically plausible for the patient's conditions and consistent with the rest of the patient's history. Each injected safety constraint mostly has a similar event (same drug) in the real record of another source patient with the same condition. Mentions of other people's conditions are common in the source records: in each condition group with at least five source patients, 58–83% of patients' records mention a family history and 57–83% mention a relative's condition. Injected content mostly serves the Safety Constraints and Entity Attribution axes (24% of questions); the two hardest axes, Longitudinal Trajectory Tracking and Cross-Department Integration, and findings (2)–(4) do not rely on it.

Below we compare the synthetic histories with the source records quantitatively.

Measurement trajectories follow the source trends and levels. For each patient's three most frequent laboratory tests, we compare the regenerated series with the source series reading by reading (Table 1). The median correlation is r = 0.94 (IQR 0.89–0.96), and 51 of 58 series have r ≥ 0.8. Regenerated means are within 10% of the source means for 45 of 58 series (median difference 4.3%). The three series with r < 0.5 (two ALT, one albumin) stay within or near the normal range throughout, where correlation is dominated by small fluctuations.

Synthetic events have real precedents. Table 2 checks, for each patient and event category, whether every synthetic event has a similar event in the real record of another source patient with the same condition. For medications and drug constraints, similar means the same drug class. For procedures, referrals and family-history events, it means the same event type: the same kind of procedure, a referral to the same specialty, or the same kind of family-history mention. During the generation process, human experts judged the matches. This holds for treatments in all 23 patients and for family-history events in all 22 patients that have them. For safety constraints it holds in 19 of 23 patients; the remaining constraints are documented contraindications [1-8].

These quantitative analyses provide strong evidence that our dataset faithfully captures clinical realism, closely matches real-world distributions of safety constraints, family histories, and measurement trajectories, and accurately reflects authentic records from other patients with the same condition.

[1]. Grice, Sophie, et al. "Omeprazole-associated atypical drug reaction with eosinophilia and systemic symptoms (DRESS) in a patient with positive In vitro diagnostic testing to multiple proton pump inhibitors." Chemical Research in Toxicology 37.9 (2024): 1484-1487.

[2]. Teles, Ariana, et al. "Paracetamol allergy: A case of a 9-year-old female with a history of atopy." Pediatric Allergy, Immunology, and Pulmonology 34.2 (2021): 80-82.

[3]. Zembowicz, Artur, et al. "Safety of cyclooxygenase 2 inhibitors and increased leukotriene synthesis in chronic idiopathic urticaria with sensitivity to nonsteroidal anti-inflammatory drugs." Archives of Dermatology 139.12 (2003): 1577-1582.

[4]. Murakami, Yoichi, et al. "Heparin-induced thrombocytopenia with anaphylactoid reaction during hemodialysis." Internal Medicine 63.16 (2024): 2321-2324.

[5]. Canalejo, Eduardo, and María S. Pacheco. "Cushing syndrome due to ritonavir–fluticasone interaction." Canadian Medical Association Journal 184.15 (2012): 1714-1714.

[6]. Badawi, Ahmed H., et al. "Cetirizine-induced acute generalized exanthematous pustulosis: a serious reaction to a commonly used drug." Dermatology online journal 20.5 (2014).

[7]. Pirson, Francoise, André Geubel, and Lilianne Marot. "Late hypersensitivity to omeprazole and other proton pump inhibitors." Acta clinica Belgica 67.4 (2012): 301-303.

[8]. Edinoff, Amber N., et al. "Zolpidem: efficacy and side effects for insomnia." Health psychology research 9.1 (2021): 24927.

Table 1. Source and regenerated values for each patient's three most frequent laboratory tests: mean (range) and correlation r across readings.

PatientLab testReadingsSource: mean (range)Regenerated: mean (range)r
P1Fasting glucose32198 (98–420)202 (79–348)0.95
P1HbA1c2910.2 (6.3–15.4)9.3 (6.2–11.4)0.76
P1LDL cholesterol22101 (71–172)102 (72–170)0.93
P2Creatinine433.26 (1.12–18.77)2.65 (0.70–13.68)0.98
P2Fasting glucose31137 (102–215)145 (104–233)0.94
P2ALT2224 (11–38)19 (5–38)0.39
P3ALT5347 (23–186)50 (5–190)0.96
P3HbA1c226.8 (6.0–7.6)6.7 (6.0–7.5)0.98
P3WBC114.2 (0.3–17.0)4.7 (1.0–21.0)0.97
P4Triglyceride39113 (62–190)118 (82–190)0.90
P4Fasting glucose2390 (79–98)84 (71–98)0.91
P4Creatinine220.94 (0.83–1.07)0.94 (0.70–1.23)0.73
P6Fasting glucose53203 (114–340)209 (106–339)0.95
P6HbA1c377.6 (6.7–10.4)7.7 (6.6–10.6)0.97
P6Triglyceride25297 (111–1081)340 (123–855)0.85
P7Potassium1124.1 (3.1–6.9)4.3 (3.1–6.6)0.90
P7Hemoglobin1058.5 (6.6–11.6)8.3 (5.1–11.8)0.86
P7Albumin844.0 (3.4–4.6)4.2 (3.2–4.9)−0.05
P8HbA1c465.5 (5.1–6.3)5.3 (4.6–6.4)0.83
P8Fasting glucose26101 (63–145)104 (61–162)0.94
P8ALT2321 (7–34)15 (6–37)0.17
P9Creatinine713.80 (0.80–10.15)4.09 (0.60–12.39)0.94
P9Fasting glucose66135 (83–264)135 (75–325)0.95
P9HbA1c447.2 (5.6–8.6)7.2 (4.9–8.7)0.95
P10Fasting glucose98104 (57–295)102 (41–336)0.96
P10Creatinine774.64 (1.20–16.16)4.45 (0.90–13.46)0.99
P10ALT6028 (7–85)30 (5–129)0.91
P11Fasting glucose42126 (77–221)130 (72–239)0.96
P11HbA1c396.9 (5.3–7.9)7.0 (4.8–7.9)0.95
P11Creatinine385.62 (1.07–13.03)5.36 (0.82–13.51)0.98
P12Hemoglobin812.6 (9.5–18.4)13.0 (9.3–17.9)0.95
P12Creatinine80.90 (0.57–1.20)0.89 (0.34–1.20)0.96
P12Uric acid55.3 (3.8–7.5)6.2 (4.7–8.6)0.94
P13Fasting glucose43268 (84–567)232 (93–346)0.87
P14ALT5163 (15–324)61 (5–285)0.94
P14Creatinine120.84 (0.54–1.20)0.77 (0.38–1.20)0.88
P14Triglyceride1086 (55–190)119 (81–190)0.97
P15WBC888.38 (2.00–44.00)8.26 (2.00–20.10)0.84
P15Hemoglobin7110.5 (7.1–15.1)10.5 (5.6–15.7)0.93
P15Creatinine661.00 (0.51–1.76)1.09 (0.28–1.90)0.88
P16CEA81.1 (0.8–1.3)2.5 (0.6–4.8)0.96
P16WBC62.75 (0.83–5.20)3.64 (0.40–9.40)0.94
P16Creatinine60.67 (0.58–0.77)0.92 (0.38–1.26)0.96
P17Creatinine481.25 (0.83–1.48)1.19 (0.69–1.90)0.70
P17Fasting glucose35140 (95–219)142 (77–238)0.93
P17HbA1c327.1 (6.4–7.7)7.2 (5.9–7.9)0.95
P18Creatinine240.55 (0.44–0.59)0.43 (0.23–0.56)0.62
P19HbA1c757.8 (5.2–9.2)7.9 (4.7–9.7)0.93
P19Post-meal glucose35210 (118–322)215 (91–331)0.93
P19Fasting glucose32254 (84–500)240 (87–343)0.89
P20HbA1c249.5 (6.7–12.2)9.5 (6.7–11.4)0.97
P20Fasting glucose21199 (129–315)210 (135–327)0.95
P21Albumin54.1 (3.0–5.8)3.8 (3.0–5.2)0.98
P22HbA1c629.1 (5.3–11.2)9.1 (4.8–11.3)0.98
P22Fasting glucose45214 (85–374)208 (93–345)0.90
P22Creatinine402.68 (0.46–11.73)2.75 (0.22–13.71)0.98
P23ALT18105 (25–226)126 (24–284)0.96
P23CEA135.0 (1.4–16.2)5.1 (0.4–14.8)0.97

Table 2. ✓ = every synthetic event in the category has a similar event in another source patient with the same condition; × = at least one does not; N/A = no event in this category.

PatientConditionsTreatmentFamily historySafety constraints
P1Musculoskeletal, Heart✓✓✓
P2Kidney, Heart, Stroke✓✓✓
P3Mental health, Musculoskeletal, Diabetes✓✓✓
P4Skin, Hyperlipidemia, Eye✓✓×
P5Cancer, Musculoskeletal✓✓✓
P6Kidney, Heart, Diabetes✓✓×
P7Kidney, Heart✓✓✓
P8Airway, Diabetes, Heart✓✓✓
P9Diabetes, Kidney, Heart✓✓✓
P10Kidney, Cancer, Diabetes✓✓✓
P11Kidney, Heart, Diabetes✓✓×
P12Cancer, Mental health✓✓✓
P13Pancreatitis, Diabetes✓N/A✓
P14Heart, Prostate, Liver✓✓✓
P15Cancer, Autoimmune, Diabetes✓✓✓
P16Cancer, Musculoskeletal, Skin✓✓✓
P17Heart, Stroke, Diabetes✓✓✓
P18Mental health, Stomach and bowel, Liver✓✓×
P19Kidney, Heart, Diabetes✓✓✓
P20Diabetes, Mental health, Heart✓✓✓
P21Cancer, Musculoskeletal✓✓✓
P22Diabetes, Musculoskeletal, Heart✓✓✓
P23Cancer, Mental health✓✓✓

W2. Do the dialogues preserve the clinical facts, and are they representative?

Fidelity is controlled at generation time and audited afterwards. We show quantitative comparison of evidence and dialogue comparison below:

Each event-chain fact is an atom with an ID. The doctor agent receives, turn by turn, the atom IDs it must convey, and each generated turn reports the IDs it actually conveys. Grounding rules forbid inventing or altering values, doses, dates and diagnoses. Small talks to add conversational realism are limited to non-clinical topics.

We used claude opus 5 to compare every event-chain fact with the dialogue of its visit and labeled it correctly expressed, omitted, or inconsistent (changed value or missing detail). Additionally, during benchmark construction, each question was answered correctly by GPT-5.6-Sol given its gold evidence visits, so a question whose required fact was lost would have been removed.

Statistics of how many facts are omitted or changed from the source event chain.

Fact fieldFactsOmittedChanged
Diagnosis30,5570.3%0.7%
Patient's own description99,8022.3%1.6%
Tests and medication98,4862.6%3.7%
Doctor's notes72,8862.1%1.0%
Doctor's plans30,5232.1%0.6%
Health advice4,2203.9%0.6%
Doctor's extra notes22,5121.9%3.2%
Vital signs4,1160.4%0.0%
All facts363,1022.1%2.0%

Clinician assessment. Clinicians rated 100 randomly sampled visit dialogues on a 5-point scale (100 ratings per question), showing the high fidelity of generated dialogues.

Rating assessment of 100 randomly sampled dialogues by clinicians.

QuestionRatings12345Mean ± SDRated 4 or 5
Q1. Fits the condition1000.0%0.0%13.0%59.0%28.0%4.15 ± 0.6387.0%
Q2. Grounded in what appeared before1001.0%0.0%3.0%23.0%73.0%4.67 ± 0.6496.0%
Q3. Doctor-Patient Role Play1000.0%0.0%6.0%60.0%34.0%4.28 ± 0.5794.0%
Q4. Alignment with real clinical interactions1000.0%4.0%17.0%54.0%25.0%4.00 ± 0.7779.0%

On the constrained prompts. The prompts in Appendix A.9 are long because they constrain content, not style. They forbid inventing clinical facts, which would corrupt the ground truth, and they keep information asymmetric as in a real visit: the patient learns results only from the doctor.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.