- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces Synthetic Hospital, an open and reproducible benchmark for evaluating language models and clinical agents on longitudinal electronic health record (EHR) tasks. The authors construct synthetic patient records from publicly available medical education materials, including USMLE-style questions, by extracting clinical concepts, grounding them in medical ontologies, and combining related cases into longitudinal patient histories. The resulting benchmark contains 1,268 synthetic patients and 5,602 encounters, and supports several tasks, including patient diagnosis, clinical summarization, evidence retrieval, and imaging indication.
The authors evaluate ten language models and conduct additional agentic experiments using a simulated EHR interface. They also assess record-level realism through a physician discrimination study and validate selected ontology-derived clinical relationships. The paper aims to provide an openly accessible alternative to real-EHR benchmarks while enabling reproducible evaluation with automatically constructed ground-truth labels.
Critical strengths
- Open and reproducible clinical evaluation environment.
The paper provides an openly accessible benchmark with explicit provenance tracking, ontology-grounded clinical concepts, and reproducible evaluation procedures. This is a meaningful contribution given the privacy and accessibility constraints associated with real EHR datasets. The ability to systematically evaluate models without requiring access to restricted patient records may facilitate future research and benchmarking.
- Unified framework for evaluating multiple clinical tasks and agentic workflows.
The benchmark incorporates several clinically motivated tasks within a unified simulated EHR environment, including support for tool-based interactions. The evaluation across ten language models, together with experiments separating agentic-loop effects from self-retrieval effects, provides useful observations about current model behavior. This framework could serve as a controlled testbed for developing and debugging EHR-interacting agents.
Critical weaknesses
- Fundamental concerns about the construct validity of the proposed longitudinal EHR benchmark.
My primary concern is whether the proposed benchmark meaningfully captures the challenges of real-world longitudinal EHR reasoning. The synthetic patients are constructed by clustering independently sourced medical examination questions based on demographic compatibility and shared diagnoses, followed by LLM-generated encounter timelines and clinical narratives.
While this produces longitudinally formatted records, it is unclear whether the resulting data captures meaningful temporal dependencies, disease progression, treatment-response dynamics, or causal relationships between encounters. Simply combining multiple clinical vignettes into a chronologically ordered patient record does not necessarily create the reasoning challenges encountered in actual longitudinal EHRs.
More importantly, the benchmark does not explicitly simulate missing information, contradictory documentation, or documentation errors, as acknowledged by the authors. These are central challenges of real-world EHR reasoning and are explicitly highlighted in the paper's motivation. Their absence raises questions about whether the benchmark evaluates clinically meaningful reasoning or primarily measures information extraction and medical knowledge retrieval from relatively clean, diagnostically informative records.
The physician realism study does not sufficiently resolve this concern. Distinguishing between five real and five synthetic records after format harmonization primarily assesses record-level plausibility, rather than whether the generated records preserve the underlying temporal, structural, and operational complexity of real clinical data.
Consequently, the paper does not convincingly establish that the proposed benchmark measures the clinical capabilities it claims to evaluate.
- Insufficient validation of clinical ground truth and real-world applicability.
A second major concern is whether the automatically constructed ground-truth labels correspond to clinically meaningful targets. The patient diagnosis reference is defined using the correct-answer diagnoses of the original medical examination questions, rather than an independently adjudicated clinical problem list.
The authors' own example illustrates this issue. In Appendix F, Patient 1973 has documented chronic conditions, including diabetes, hypertension, and chronic kidney disease, yet the reference problem list contains no chronic diagnoses. Meanwhile, acute diagnoses from encounters separated by several months remain classified as active diagnoses. Although the chart-neutral scoring rule avoids penalizing additional documented diagnoses, it does not resolve the potential mismatch between the benchmark's positive targets and clinically appropriate longitudinal problem-list maintenance.
This raises concerns about whether high benchmark scores necessarily reflect clinically correct behavior. Deterministic label construction ensures reproducibility, but does not by itself guarantee clinical validity.
Furthermore, the paper provides no direct evidence that model performance on Synthetic Hospital correlates with performance on real EHR tasks. The benchmark may distinguish models within its synthetic environment, but whether these distinctions translate to actual clinical workflows remains unknown.
Without independent clinical validation of the constructed labels and some evidence of transferability to real EHRs, I find it difficult to assess the practical significance of the reported model rankings and agentic performance results.
Questions for authors
Q1. Evidence of real-world transferability.
Have the authors evaluated whether model performance or rankings on Synthetic Hospital correlate with those obtained on real EHR-based clinical tasks? Even a relatively small, clinician-adjudicated real-EHR evaluation would help establish whether this benchmark provides meaningful information about real-world clinical capabilities. Without such evidence, what supports the practical interpretation of the reported model rankings?
Q2. Validation of longitudinal clinical coherence.
Given that longitudinal patients are assembled by clustering independently sourced medical examination vignettes, how do the authors establish that the resulting encounter sequences represent clinically meaningful longitudinal trajectories rather than collections of loosely related diagnostic cases? Is there any quantitative or physician-based evaluation of temporal consistency, disease progression, and inter-encounter dependencies? Additionally, would model performance change substantially if encounter order were shuffled or longitudinal context were removed?
Q3. Clinical validity of ground-truth problem lists.
How do the authors justify defining the reference problem list primarily through the correct-answer diagnoses of the original examination questions? For instance, in Patient 1973, why are documented chronic diseases excluded from the reference while historical acute diagnoses remain classified as active? Have the authors compared the constructed ground truth against independently physician-annotated problem lists to determine whether the scoring procedure rewards clinically appropriate predictions?
Q4. Clinical meaningfulness of the summarization metric.
Table 3 reports a substantially lower finding-level summarization score for physicians than for language models, although these results are not evaluated on identical sample sizes. Could the authors clarify why physician performance is so low under this metric? Does the evaluation reward exhaustive reproduction of predefined findings rather than clinically useful summarization? A matched comparison using physician-adjudicated summary quality would help establish the clinical validity of this metric.
Responses to Reviewer 3
Q1: My primary concern is whether the proposed benchmark meaningfully captures the challenges of real-world longitudinal EHR reasoning. […]
A1: Records rarely capture physicians' reasoning [1], so we used physician realism judgments and benchmark tasks. Physicians agreed with up to 89% of reference diagnoses. Agreement on diagnoses and evidence supports relevance; matching physician reasoning remains future work.
Q2: […] the benchmark does not explicitly simulate missing information, contradictory documentation, or documentation errors […]
A2: We randomly removed 10%, 25%, or 50% of encounters (F1 below). Gemini 3.1 replaced unavailable GPT 5.3. Contradictions and documentation errors remain untested.
Q3: […] Distinguishing between five real and five synthetic records after format harmonization primarily assesses record-level plausibility […]
A3: Standardized formatting removed format cues. Physician accuracy was 53% (95% interval 43%–63%; p = 0.62), consistent with chance. We will replace “indistinguishable” with this limited plausibility finding.
Q4: […] The patient diagnosis reference is defined using the correct-answer diagnoses of the original medical examination questions, rather than an independently adjudicated clinical problem list.
A4: Independent physician adjudication is the gold standard. Physician-authored diagnoses are ontology-mapped, assembled deterministically, and traceable; non-diagnostic answers are excluded. The graph recovered 111/119 preregistered physician-defined relationships (93%); eight misses were excluded epidemiologic associations. This does not validate each problem list. Three reviewers × three minutes × 5,602 encounters require ~840 physician-hours before consensus. We added a stratified-sample audit to Limitations.
Q5: […] acute diagnoses from encounters separated by several months remain classified as active diagnoses. […]
A5: Resolved diagnoses remain targets because we test recovery across the record. We will rename the task “longitudinal diagnosis ascertainment” and the field “encounter diagnoses.” Clearer prompts change recall by at most 0.02 and lower precision; wording alone does not explain misses.
Q6: Furthermore, the paper provides no direct evidence that model performance on Synthetic Hospital correlates with performance on real EHR tasks. […]
A6: We have not measured correlation with real-EHR performance. Physician agreement supports clinical relevance but does not establish correlation (A1).
Q7: […] Have the authors evaluated whether model performance or rankings on Synthetic Hospital correlate with those obtained on real EHR-based clinical tasks? […]
A7: See A6.
Q8: […] Is there any quantitative or physician-based evaluation of temporal consistency, disease progression, and inter-encounter dependencies?
A8: Across six conditions varying order/context, shuffling changed diagnosis F1 by −0.03 to +0.01. Keeping only the final encounter reduced summary recall by 40%–57%; also removing its problem list reduced diagnosis F1 by 0.18–0.24. Earlier context matters; primary scores do not assess order. We will report temporal placement separately.
Q9: Additionally, would model performance change substantially if encounter order were shuffled or longitudinal context were removed?
A9: See A8.
Q10: […] in Patient 1973, why are documented chronic diseases excluded from the reference while historical acute diagnoses remain classified as active?
A10: Patient 1973's resolved appendicitis remains a target; profile chronic conditions neither earn credit nor incur penalties. We test cumulative recovery, will replace “active” (A5), and report documented-condition recall separately.
Q11: Have the authors compared the constructed ground truth against independently physician-annotated problem lists […]?
A11: See A4.
Q12: Clinical meaningfulness of the summarization metric. […] Could the authors clarify why physician performance is so low under this metric?
A12: Physicians focused on recent findings and had no graph hints. On the same 13 patients, truncating model summaries to the physician mean of 52 words lowers scores to 0.12–0.21 versus 0.05 for physicians. We will report lengths and matched scores, use zero-shot prompts, and remove the cap.
Q13: Does the evaluation reward exhaustive reproduction of predefined findings rather than clinically useful summarization? […]
A13: Coverage scores may favor length and do not assess usefulness. We will call it “key-finding recall” and report fixed-length comparisons: scores are 0.188–0.271 for the first 52 words versus 0.385–0.550 for full summaries. Next, blinded physicians will assess completeness, correctness, and usefulness at equal lengths.