acceptodds
Under review as a conference paper at ICLR 2027

Synthetic Hospital: An Open, Verifiable, Physician-Validated Longitudinal EHR Benchmark

Abstract

Frontier language models are rarely used in clinical workflows because the realistic, longitudinal benchmarks needed to develop them are scarce. Real electronic health record (EHR) data cannot be openly shared due to privacy, ethics or data use issues and it does not contain verifiable ground truth since the chart records only reflect what clinicians documented. We introduce Synthetic Hospital, an open, fully synthetic, fact-grounded longitudinal EHR benchmark that resolves the open sharing and verifiable ground truth barriers. Built entirely from public medical-education material with no protected health information, it comprises 1,268 longitudinal patients and 5,602 encounters, where every diagnosis, finding, and temporal relation is grounded in standard ontologies (ICD-10-CM, SNOMED CT, LOINC) and with a complete provenance chain back to its source medical education material. Synthetic Hospital is served through a simulated hospital record system that mirrors real EHR infrastructure (standard interoperability APIs, role-based access and function-calling interface). In a blinded review, physicians distinguished its records from real patient charts at near-chance rates (53%). Across 10 frontier and open models, none approaches ceiling: the best model reconstructs a patient's longitudinal problem list with a severity-weighted F1 of 0.73, level with the mean of seven physicians on a matched subset but well below the best of them (0.89), and misses roughly half of clinically relevant findings when summarizing a chart. Overall, these results highlight that Synthetic Hospital is a difficult and realistic test of clinical AI performance.

Primary area:
datasets and benchmarks
Keywords:
clinical AI benchmark, electronic health records, synthetic data, longitudinal patient records, LLM evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PfcnOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces Synthetic Hospital, an open and reproducible benchmark for evaluating language models and clinical agents on longitudinal electronic health record (EHR) tasks. The authors construct synthetic patient records from publicly available medical education materials, including USMLE-style questions, by extracting clinical concepts, grounding them in medical ontologies, and combining related cases into longitudinal patient histories. The resulting benchmark contains 1,268 synthetic patients and 5,602 encounters, and supports several tasks, including patient diagnosis, clinical summarization, evidence retrieval, and imaging indication.

The authors evaluate ten language models and conduct additional agentic experiments using a simulated EHR interface. They also assess record-level realism through a physician discrimination study and validate selected ontology-derived clinical relationships. The paper aims to provide an openly accessible alternative to real-EHR benchmarks while enabling reproducible evaluation with automatically constructed ground-truth labels.

Critical strengths
  1. Open and reproducible clinical evaluation environment.

The paper provides an openly accessible benchmark with explicit provenance tracking, ontology-grounded clinical concepts, and reproducible evaluation procedures. This is a meaningful contribution given the privacy and accessibility constraints associated with real EHR datasets. The ability to systematically evaluate models without requiring access to restricted patient records may facilitate future research and benchmarking.

  1. Unified framework for evaluating multiple clinical tasks and agentic workflows.

The benchmark incorporates several clinically motivated tasks within a unified simulated EHR environment, including support for tool-based interactions. The evaluation across ten language models, together with experiments separating agentic-loop effects from self-retrieval effects, provides useful observations about current model behavior. This framework could serve as a controlled testbed for developing and debugging EHR-interacting agents.

Critical weaknesses
  1. Fundamental concerns about the construct validity of the proposed longitudinal EHR benchmark.

My primary concern is whether the proposed benchmark meaningfully captures the challenges of real-world longitudinal EHR reasoning. The synthetic patients are constructed by clustering independently sourced medical examination questions based on demographic compatibility and shared diagnoses, followed by LLM-generated encounter timelines and clinical narratives.

While this produces longitudinally formatted records, it is unclear whether the resulting data captures meaningful temporal dependencies, disease progression, treatment-response dynamics, or causal relationships between encounters. Simply combining multiple clinical vignettes into a chronologically ordered patient record does not necessarily create the reasoning challenges encountered in actual longitudinal EHRs.

More importantly, the benchmark does not explicitly simulate missing information, contradictory documentation, or documentation errors, as acknowledged by the authors. These are central challenges of real-world EHR reasoning and are explicitly highlighted in the paper's motivation. Their absence raises questions about whether the benchmark evaluates clinically meaningful reasoning or primarily measures information extraction and medical knowledge retrieval from relatively clean, diagnostically informative records.

The physician realism study does not sufficiently resolve this concern. Distinguishing between five real and five synthetic records after format harmonization primarily assesses record-level plausibility, rather than whether the generated records preserve the underlying temporal, structural, and operational complexity of real clinical data.

Consequently, the paper does not convincingly establish that the proposed benchmark measures the clinical capabilities it claims to evaluate.

  1. Insufficient validation of clinical ground truth and real-world applicability.

A second major concern is whether the automatically constructed ground-truth labels correspond to clinically meaningful targets. The patient diagnosis reference is defined using the correct-answer diagnoses of the original medical examination questions, rather than an independently adjudicated clinical problem list.

The authors' own example illustrates this issue. In Appendix F, Patient 1973 has documented chronic conditions, including diabetes, hypertension, and chronic kidney disease, yet the reference problem list contains no chronic diagnoses. Meanwhile, acute diagnoses from encounters separated by several months remain classified as active diagnoses. Although the chart-neutral scoring rule avoids penalizing additional documented diagnoses, it does not resolve the potential mismatch between the benchmark's positive targets and clinically appropriate longitudinal problem-list maintenance.

This raises concerns about whether high benchmark scores necessarily reflect clinically correct behavior. Deterministic label construction ensures reproducibility, but does not by itself guarantee clinical validity.

Furthermore, the paper provides no direct evidence that model performance on Synthetic Hospital correlates with performance on real EHR tasks. The benchmark may distinguish models within its synthetic environment, but whether these distinctions translate to actual clinical workflows remains unknown.

Without independent clinical validation of the constructed labels and some evidence of transferability to real EHRs, I find it difficult to assess the practical significance of the reported model rankings and agentic performance results.

Questions for authors

Q1. Evidence of real-world transferability.

Have the authors evaluated whether model performance or rankings on Synthetic Hospital correlate with those obtained on real EHR-based clinical tasks? Even a relatively small, clinician-adjudicated real-EHR evaluation would help establish whether this benchmark provides meaningful information about real-world clinical capabilities. Without such evidence, what supports the practical interpretation of the reported model rankings?

Q2. Validation of longitudinal clinical coherence.

Given that longitudinal patients are assembled by clustering independently sourced medical examination vignettes, how do the authors establish that the resulting encounter sequences represent clinically meaningful longitudinal trajectories rather than collections of loosely related diagnostic cases? Is there any quantitative or physician-based evaluation of temporal consistency, disease progression, and inter-encounter dependencies? Additionally, would model performance change substantially if encounter order were shuffled or longitudinal context were removed?

Q3. Clinical validity of ground-truth problem lists.

How do the authors justify defining the reference problem list primarily through the correct-answer diagnoses of the original examination questions? For instance, in Patient 1973, why are documented chronic diseases excluded from the reference while historical acute diagnoses remain classified as active? Have the authors compared the constructed ground truth against independently physician-annotated problem lists to determine whether the scoring procedure rewards clinically appropriate predictions?

Q4. Clinical meaningfulness of the summarization metric.

Table 3 reports a substantially lower finding-level summarization score for physicians than for language models, although these results are not evaluated on identical sample sizes. Could the authors clarify why physician performance is so low under this metric? Does the evaluation reward exhaustive reproduction of predefined findings rather than clinically useful summarization? A matched comparison using physician-adjudicated summary quality would help establish the clinical validity of this metric.

1 comment
AuthorsOpenReview ↗

Responses to Reviewer 3

Q1: My primary concern is whether the proposed benchmark meaningfully captures the challenges of real-world longitudinal EHR reasoning. […]

A1: Records rarely capture physicians' reasoning [1], so we used physician realism judgments and benchmark tasks. Physicians agreed with up to 89% of reference diagnoses. Agreement on diagnoses and evidence supports relevance; matching physician reasoning remains future work.

Q2: […] the benchmark does not explicitly simulate missing information, contradictory documentation, or documentation errors […]

A2: We randomly removed 10%, 25%, or 50% of encounters (F1 below). Gemini 3.1 replaced unavailable GPT 5.3. Contradictions and documentation errors remain untested.

Model50% removedNo labs
Gemini 3.10.6130.682
Mistral Large0.6180.686
Llama 4 Scout0.5000.519

Q3: […] Distinguishing between five real and five synthetic records after format harmonization primarily assesses record-level plausibility […]

A3: Standardized formatting removed format cues. Physician accuracy was 53% (95% interval 43%–63%; p = 0.62), consistent with chance. We will replace “indistinguishable” with this limited plausibility finding.

Q4: […] The patient diagnosis reference is defined using the correct-answer diagnoses of the original medical examination questions, rather than an independently adjudicated clinical problem list.

A4: Independent physician adjudication is the gold standard. Physician-authored diagnoses are ontology-mapped, assembled deterministically, and traceable; non-diagnostic answers are excluded. The graph recovered 111/119 preregistered physician-defined relationships (93%); eight misses were excluded epidemiologic associations. This does not validate each problem list. Three reviewers × three minutes × 5,602 encounters require ~840 physician-hours before consensus. We added a stratified-sample audit to Limitations.

Q5: […] acute diagnoses from encounters separated by several months remain classified as active diagnoses. […]

A5: Resolved diagnoses remain targets because we test recovery across the record. We will rename the task “longitudinal diagnosis ascertainment” and the field “encounter diagnoses.” Clearer prompts change recall by at most 0.02 and lower precision; wording alone does not explain misses.

Q6: Furthermore, the paper provides no direct evidence that model performance on Synthetic Hospital correlates with performance on real EHR tasks. […]

A6: We have not measured correlation with real-EHR performance. Physician agreement supports clinical relevance but does not establish correlation (A1).

Q7: […] Have the authors evaluated whether model performance or rankings on Synthetic Hospital correlate with those obtained on real EHR-based clinical tasks? […]

A7: See A6.

Q8: […] Is there any quantitative or physician-based evaluation of temporal consistency, disease progression, and inter-encounter dependencies?

A8: Across six conditions varying order/context, shuffling changed diagnosis F1 by −0.03 to +0.01. Keeping only the final encounter reduced summary recall by 40%–57%; also removing its problem list reduced diagnosis F1 by 0.18–0.24. Earlier context matters; primary scores do not assess order. We will report temporal placement separately.

Q9: Additionally, would model performance change substantially if encounter order were shuffled or longitudinal context were removed?

A9: See A8.

Q10: […] in Patient 1973, why are documented chronic diseases excluded from the reference while historical acute diagnoses remain classified as active?

A10: Patient 1973's resolved appendicitis remains a target; profile chronic conditions neither earn credit nor incur penalties. We test cumulative recovery, will replace “active” (A5), and report documented-condition recall separately.

Q11: Have the authors compared the constructed ground truth against independently physician-annotated problem lists […]?

A11: See A4.

Q12: Clinical meaningfulness of the summarization metric. […] Could the authors clarify why physician performance is so low under this metric?

A12: Physicians focused on recent findings and had no graph hints. On the same 13 patients, truncating model summaries to the physician mean of 52 words lowers scores to 0.12–0.21 versus 0.05 for physicians. We will report lengths and matched scores, use zero-shot prompts, and remove the cap.

Q13: Does the evaluation reward exhaustive reproduction of predefined findings rather than clinically useful summarization? […]

A13: Coverage scores may favor length and do not assess usefulness. We will call it “key-finding recall” and report fixed-length comparisons: scores are 0.188–0.271 for the first 52 words versus 0.385–0.550 for full summaries. Next, blinded physicians will assess completeness, correctness, and usefulness at equal lengths.

Reviewer_xt5KOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Synthetic Hospital, a fully synthetic longitudinal electronic health record (EHR) benchmark constructed from publicly available medical educational materials. The framework extracts clinical concepts from medical examination questions, maps them to standard medical ontologies, and organizes them into a knowledge graph. Independent clinical cases are then grouped into longitudinal patient records based on shared diagnoses and demographic compatibility, with LLMs used to generate patient profiles and encounter timelines.

The resulting benchmark contains 1,268 synthetic patients and 5,602 encounters, supporting clinical diagnosis, summarization, evidence retrieval, and imaging indication tasks. The authors evaluate the realism of the generated records through a blinded physician study, examine the coverage of clinical relationships in the knowledge graph, and benchmark ten language models and several agentic configurations.

Critical strengths
  1. The paper addresses a relevant challenge in clinical AI evaluation: the limited availability of openly accessible longitudinal EHR benchmarks. The integration of medical educational materials, standard clinical ontologies, and knowledge graph-based provenance provides a structured and reproducible approach to benchmark construction. Deriving evaluation labels from structured data rather than directly from generated narratives is a useful design choice.

  2. Broad evaluation with informative agentic comparisons. The benchmark covers several clinically relevant tasks and evaluates ten language models. The additional comparison between full-context inference, full-context agents, and self-retrieving agents is particularly useful. The results suggest that autonomous retrieval can improve longitudinal diagnosis while introducing additional costs for tasks where the relevant clinical context is already available.

Critical weaknesses
  1. The benchmark constructs longitudinal records by clustering independent medical examination cases based on shared diagnoses and demographic compatibility. However, examination questions contain selected diagnostic findings rather than complete clinical encounters. Such clustering does not guarantee consistency in disease progression, treatment history, or patient state transitions. Furthermore, the physician study (53% accuracy) primarily evaluates whether records appear realistic, rather than whether the complete longitudinal trajectories are clinically coherent. Therefore, the current evidence is insufficient to establish the clinical validity of the constructed longitudinal EHRs.

  2. The ground truth for patient diagnosis includes only the correct-answer diagnoses from source questions, excluding secondary diagnoses and other documented conditions. More importantly, the evaluation does not explicitly assess whether findings occur at the correct encounters or whether clinical states change consistently over time. Consequently, high scores may reflect the recovery of selected diagnoses and findings rather than genuine longitudinal clinical reasoning, raising concerns about the benchmark's construct validity.

  3. Temporal consistency is particularly important when independent clinical cases are combined into a longitudinal patient record. Although the framework generates encounter timelines and applies chronological constraints, these checks do not establish whether symptoms, diagnoses, treatments, and medications evolve consistently over time. The paper does not report a systematic evaluation or quantitative evidence of such temporal consistency. This raises concerns about whether the constructed records represent coherent longitudinal clinical trajectories rather than simply chronologically arranged clinical cases.

Questions for authors
  1. How is clinical consistency across encounters validated beyond shared diagnoses and demographic compatibility? Have physicians systematically evaluated complete patient trajectories for disease progression, treatment consistency, medication changes, and contradictions between encounters? If so, what are the observed error rates?

  2. How is the claim of complete and verifiable ground truth justified when secondary diagnoses and other documented conditions are excluded from the patient diagnosis reference? How does the cap of 20 key findings affect the evaluation of patients with multiple encounters, particularly when findings from later encounters are excluded?

  3. Do the current scoring procedures penalize diagnoses or findings assigned to incorrect encounters, incorrect temporal ordering, or inconsistent changes in clinical state? Could the authors provide encounter-level temporal consistency metrics or an experiment testing whether changing encounter order affects model performance?

  4. Were the physicians explicitly instructed to assess cross-encounter clinical consistency, or only to distinguish synthetic records from real ones? Additional physician evaluation focused on longitudinal coherence would be important for supporting the benchmark's central claim.

1 comment
AuthorsOpenReview ↗

Responses to Reviewer 2

Q1: […] Such clustering does not guarantee consistency in disease progression, treatment history, or patient state transitions.

A1: We group related cases, preserve expert keys, share diagnoses and a fixed patient profile, and connect histories without adding diagnostic evidence. Checks cover all 1,268 patients (A5). These preserve documented information; disease progression and treatment response remain unvalidated.

Q2: […] the physician study (53% accuracy) primarily evaluates whether records appear realistic, rather than whether the complete longitudinal trajectories are clinically coherent.

A2: Complete-chart realism judgments provide some evidence of internal consistency but do not validate disease progression. We will clarify this scope and add a dedicated physician audit in the next version.

Q3: The ground truth for patient diagnosis includes only the correct-answer diagnoses from source questions, excluding secondary diagnoses and other documented conditions.

A3: Expert keys define targets independently of note generation. Additional documented conditions neither earn credit nor incur penalties. We will add their recall to the appendix: chronic conditions, 0.766–0.891; secondary diagnoses, 0.205–0.377; combined, 0.346–0.502. Adding secondary targets would complicate evaluation but may help if models saturate the benchmark.

Q4: […] the evaluation does not explicitly assess whether findings occur at the correct encounters or whether clinical states change consistently over time.

A4: Primary scores measure diagnosis recovery. Separately, models correctly date 74%–84% of matched diagnoses and order 73%–79% of diagnosis pairs; 42%–62% of reference diagnoses are both recovered and correctly dated. We will report these metrics and narrow our longitudinal-reasoning claims. Clinical state changes remain unscored.

Q5: […] The paper does not report a systematic evaluation or quantitative evidence of such temporal consistency.

A5: Across 1,268 patients, all chronological checks pass. Later notes retain earlier diagnoses in 98.3%, chronic conditions in 98.1%, and medications in 99.96% of checks; age agrees with the timeline in 97.0%. We will clarify that these assess information preservation, not disease progression, and release the script.

Q6: […] Have physicians systematically evaluated complete patient trajectories for disease progression, treatment consistency, medication changes, and contradictions between encounters? […]

A6: Seven physicians reviewed complete real and synthetic charts: accuracy was 56% across 84 judgments (95% CI 45–67%). A physician reviewed generated narratives; the graph recovered 93% of 119 preregistered physician-defined diagnosis relationships. These checks do not establish trajectory correctness. Labels derive deterministically from the graph, independently of narratives. Auditing all four dimensions would take an estimated 100–200 physician-hours for the 200-patient split. We note this limitation and plan a multi-institution audit.

Q7: How is the claim of complete and verifiable ground truth justified when secondary diagnoses and other documented conditions are excluded from the patient diagnosis reference?

A7: We will limit completeness claims to expert-keyed targets and report recall of other documented conditions separately (A3).

Q8: How does the cap of 20 key findings affect the evaluation […] particularly when findings from later encounters are excluded?

A8: We will remove the cap favoring early encounters. Uncapped scores rise by 0.008–0.019, with rankings largely unchanged (Spearman correlation 0.988). Capped results will remain in the appendix.

Q9: Do the current scoring procedures penalize diagnoses or findings assigned to incorrect encounters, incorrect temporal ordering, or inconsistent changes in clinical state?

A9: Primary diagnosis and summary scores do not penalize incorrect placement or order. We will add date accuracy and ordering agreement (A4). Retrieval and imaging use encounter-specific information; clinical state changes remain unscored.

Q10: Could the authors provide encounter-level temporal consistency metrics or an experiment testing whether changing encounter order affects model performance?

A10: Across six order/context conditions, shuffling had small, mixed effects. Keeping only the final encounter reduced summary recall by 40%–57%; also removing its problem list reduced diagnosis F1 by 0.18–0.24. Earlier context matters; temporal placement is assessed separately (A4).

Q11: Were the physicians explicitly instructed to assess cross-encounter clinical consistency, or only to distinguish synthetic records from real ones? […]

A11: Physicians classified complete charts as real or synthetic without explicit consistency instructions, to avoid biasing their attention. Their explanations cited progression and inconsistencies (A6).

Reviewer_zSWqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors introduce Synthetic Hospital, a synthetic electronic health record (EHR)-based benchmark built using medical education material. The method was evaluated using a Turing test with physicians (Section 4.1) and compared ontology-based (Section 4.2). While the work is interesting and the human physician evaluations demonstrate clear promise, the manuscript conflates clinical terminologies (any type of vocabulary that permit a range of possible relationships) with ontologies (which are formal representations). Given that ontology-grounding via a knowledge graph to prevent manual annotation is the core differentiating element of this work, the authors should take care in describing what specific ontological aspects are used (from Appendix D it seems like most of it was mapped to SNOMED-CT). Additionally, I have concerns about the measurement validity of this benchmark, as it emphasizes realism on clinical status but not on what makes real-world EHR data difficult (missingness, sparsity, medical errors, etc.) Therefore, I recommend rejection for this work and encourage the authors to resubmit to an informatics venue.

Critical strengths
  • Problem setting. The authors use medical education materials over EHR data to generate their benchmark. It’s no great secret that clinically valid EHR data is scarce and I applaud the authors for taking an unique approach for data generation.
Critical weaknesses
  • Terminologies versus ontologies. The authors use three terminologies that they refer to as ontologies (e.g. “grounded in standard ontologies (ICD-10-CM, SNOMED CT, LOINC).”) Of these, only SNOMED CT is generally considered an ontology in clinical informatics (although even this is debated). ICD-10-CM is a statistical (clinical) classification system, while LOINC is a controlled vocabulary (unless the authors are using the LOINC Ontology, which is different from LOINC–from Appendix D it seems like the authors elected to use standard LOINC). There are differences between clinical terminologies and ontologies (namely, ontologies are formal representations that enable automated reasoning, see [1,2]). The authors should clarify this and any limitations that may occur in using ICD-10-CM or LOINC relationships, as I was unable to parse this after reading Appendix D.
  • Measurement validity. The motivation the authors use in creating their benchmark raises the question of measurement validity. As the authors (and their citations) point out, real world EHR data is messy, often missing, and a reflection of billing and care practices as much as it is a log of clinical care itself. If a model is performant on Synthetic Hospital, what does that mean for its performance on actual real world EHR data? Leading EHR models such as Delphi [3] and ETHOS [4] explicitly design around missingness as a core challenge. The authors surface this in the Conclusions and Limitations section of their paper but they do not justify how this does not impact the utility and validity of their benchmark.

[1] Ivanović, Mirjana, and Zoran Budimac. "An overview of ontologies and data resources in medical domains." Expert Systems with Applications 41.11 (2014): 5158-5166.

[2] de Keizer, Nicolette F., Ameen Abu-Hanna, and Johanna HM Zwetsloot-Schonk. "Understanding terminological systems I: Terminology and typology." Methods of information in medicine 39.01 (2000): 16-21.

[3] Shmatko, Artem, et al. "Learning the natural history of human disease with generative transformers." Nature 647.8088 (2025): 248-256.

[4] Renc, Pawel, et al. "Zero shot health trajectory prediction using transformer." NPJ digital medicine 7.1 (2024): 256.

Questions for authors

Please see the weaknesses section.

1 comment
AuthorsOpenReview ↗

Responses to Reviewer 1

Q1: [...] The authors use three terminologies that they refer to as ontologies[...]only SNOMED CT is generally considered an ontology in clinical informatics [...] ICD-10-CM is a statistical (clinical) classification system, while LOINC is a controlled vocabulary [..]There are differences between clinical terminologies and ontologies[..]

A1: We will update our draft to be more precise about what SNOMET, ICD-10-CM and LOINC is. A good way to think about why we use these systems is not to provide a strict ontology in the strict sense of being logically sound. We use the systems to provide hierarchical categorical structure to our data that we can use for (1) LLM generation guided by structure, (2) having rich labels available that can be used for reinforcement learning and (3) creation of the existing and new complex tasks.

Q2: The authors should clarify [...]limitations that may occur in using ICD-10-CM or LOINC relationships[...]

A2: Standard codes that define categorical diagnostic detail are essential for our method to be able to create datasets and have reliable scoring. We find these standard codes to be effective and we found it difficult to use other standardizations to achieve reliable diagnostic level scoring/labels.

But there are limitations that we will discuss in the limitation section in more detail:

  • Because ICD chapters mix body systems and causes of disease, a disease affecting several systems can be assigned too narrowly (Appendix F).

  • Tabular List notes are coding instructions and applying them to an entire three-character category extends them beyond the original code.

  • We use LOINC for labs and quality is limited due to findings being qualitative composites with a coverage of only 28.9% of laboratory values.

  • Diagnosis-to-diagnosis links affect only the relevance categories used in specialty-conditioned summarization.

Q3: The motivation the authors use in creating their benchmark raises the question of measurement validity. [...] real world EHR data is messy, often missing, and a reflection of billing and care practices as much as it is a log of clinical care itself.

A3: We worked on a method where we can programmatically increase the “messiness” of the data. However, we left this for our follow-up work because (1) models still make substantial errors without the added messiness and (2) to do the proper science of this, we would need to measure the realism of our EHR data at different levels. We would need to conduct studies with about 50 physicians for proper science. We did not have the resources for this for our initial work. As such, this undertaking is currently out of scope of this work.

Q4: If a model is performant on Synthetic Hospital, what does that mean for its performance on actual real world EHR data?

A4: There are multiple ways to verify that performance on our benchmark can be generalized to the real setting. First, the direct way of testing trained models in a real clinical setting / on real data. But this is very difficult and expensive. Instead, we opted for a more indirect test of (1) verifying data quality with real physicians, (2) having real physicians do our tasks and see if their responses are similar to our labels. We find both (1) and (2) are well supported: physicians cannot tell the difference between real and synthetic patients and physicians agree with up to 89% of the diagnosis ground truth. As such, we have good reason to believe that model performance on the benchmark generalizes reasonably well to real data because this is what we see in real physicians who handle such data in their day-to-day job.

Q5: Leading EHR models such as Delphi [3] and ETHOS [4] explicitly design around missingness[...]The authors [...] do not justify how this does not impact the utility and validity of their benchmark.

A5: This is a great point and we ran new experiments to analyze this. Note that we replaced GPT 5.3 for Gemini 3.1 as the frontier model option because the former was no longer available. We found that including missingness only slightly changed outcomes and that even in its current form the benchmark seems to offer high utility (also see A4). See the table below for the experimental results. We include missingness by randomly removing 10%, 25%, and 50% of patient encounters across the longitudinal charts:

ConditionGemini 3.1Mistral LargeLlama 4 Scout
Patient diagnosis, chart-neutral F1
Original chart0.6710.6690.554
10% of encounters removed0.655 (-0.016)0.674 (+0.005)0.543 (-0.011)
25% of encounters removed0.649 (-0.022)0.643 (-0.026)0.512 (-0.042)
50% of encounters removed0.613 (-0.058)0.618 (-0.051)0.500 (-0.054)
Laboratory sections removed0.682 (+0.011)0.686 (+0.017)0.519 (-0.035)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.