acceptodds
Under review as a conference paper at ICLR 2027

PhenoPatient: Clinically Plausible and Diverse De Novo Virtual Patients Aligned with Real-World Phenotype Distributions

Abstract

Interactive virtual patients support medical-AI training and evaluation, but many existing simulators depend on real clinical records. Limited shareability, privacy and data-use restrictions, and scarce rare-disease records constrain their coverage. De novo simulation removes one-to-one record dependence but creates a central tension between individual coherence and cohort diversity: maximizing typicality can produce stereotyped cases, whereas expanding coverage can introduce within-patient contradictions. We introduce PhenoPatient, which reconciles these objectives while targeting real-world disease-phenotype distributions. A medical literature retrieval engine supports a severity-conditioned phenotype atlas; probabilistic sampling, cross-modal correction, and an immutable, rule-checked ledger produce diverse, coherent patients. Evaluation spans individual-level plausibility, population-level plausibility, and within-disease diversity. Across 1,000 completed encounters per implementation and 50 diseases, PhenoPatient achieves individual-level plausibility comparable to record-grounded simulators, with higher observed scores on most population-level measures and all five within-disease coverage and diversity measures. Under fixed interaction protocols, it yields the largest observed GPT-5.6 Sol–Qwen3.8-27B gaps in broad-category and specific-disease diagnosis, consistent with heterogeneous cases exposing doctor-model differences. Replacing GPT-5.6 Sol with Qwen3.8-27B for patient dialogue produces the smallest mean quality decline among evaluated simulators, consistent with ledger-supported smaller-model operation. Blinded physicians assign the highest reported mean plausibility, logical-consistency, and patient-likeness ratings. These findings support scalable, disease-diverse cases for model evaluation, training-data construction, and controlled clinical-reasoning training.

Primary area:
datasets and benchmarks
Keywords:
virtual patients, clinical simulation, large language models, synthetic patient generation, phenotype distributions, clinical reasoning evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TfuzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PhenoPatient, a method for generating virtual patients without real records. An LLM proposes candidate phenotypes in five modalities for each disease, and a commercial retrieval engine (JD Zhiyi) adds missing candidates and literature-derived prevalence and frequency evidence, which together form a severity-conditioned phenotype atlas.

Critical strengths

The comparison is carefully controlled. All seven simulators share the same disease allocation and case slots, the same roles, language and interaction budget, and the doctor-model and patient-model experiments reuse identical frozen panels, so differences between rows are not explained by different patient samples. The paper also reports what other papers often omit, including the 35% non-convergence rate of candidate patients, operational definitions for every metric, a fully traced example case, and Holm-adjusted paired tests for the physician study. This matters for my assessment because the infrastructure would support a sound comparison if the reference targets were independent.

A second strength is the framing. Separating individual-level plausibility, population-level agreement and within-disease diversity makes explicit a trade-off that patient simulators usually leave implicit: typical cases are coherent but uniform, while broad sampling creates contradictions. Record-free generation also addresses a real constraint on sharing clinical data.

Critical weaknesses

My main concern is that the population-level and diversity results, which carry the paper's central claim of alignment with real-world phenotype distributions, are largely circular. The prevalence, phenotype-frequency, course-length and numeric targets used for scoring come from the same literature retrieval that parameterized PhenoPatient's atlas, and the coverage denominators are candidate sets derived from the same evidence. PhenoPatient therefore samples from the distributions it is scored against, while the record-grounded simulators never saw them. Real MIMIC-IV patients score far below the synthetic cohort on these measures, which shows that the metrics capture conformity to the target specification more than realism. Differences between the literature populations and US inpatients, and the Chinese-language rendering of English records, would penalize the record-grounded simulators further. The ablations point the same way, since Direct Values, which stops sampling from literature ranges, loses 11.61 points of IQR matching. The metrics also test marginal rates only. Phenotypes are drawn independently given disease and severity, so their co-occurrence is unconstrained, and the case study's patient with mild disease activity carries 51 positive findings. An independent reference is needed, such as phenotype frequencies observed in MIMIC-IV for the same diseases, held-out literature, or HPO frequency annotations, together with a check of joint co-occurrence.

The evidence for individual plausibility has its own problem: the judges may prefer generated cases, and the diagnosis results suggest those cases are easier than real ones. GPT-5.5 plays the patient and the physician and also scores all four judged plausibility metrics, with no validation of the judge against clinicians. The LLM-only baseline, which invents a whole case from age, sex and diagnosis, receives the highest symptom and sign plausibility of all seven simulators and higher course plausibility than every simulator built from real records, which suggests the judge rewards textbook typicality. The physician study shows the same pattern and adds a reliability problem. Krippendorff's alpha lies between 0.03 and 0.10, 65% of all ratings are 4, and blinding is hard to maintain when record-derived cases visibly differ from generated ones. In the doctor experiment, the three strongest models reach 98 to 99% broad-category accuracy on PhenoPatient cases, against 71 to 81% on the record-grounded simulators. The largest Sol-Qwen gap therefore comes from the weaker model while the stronger one is at ceiling, and the differences between gaps (25.2 against 22.0, and 35.0 against 31.4) come from single runs with no uncertainty estimates.

Questions for authors

Some clarifications could change my assessment. How do PhenoPatient's revealed phenotype frequencies compare with an independent reference, such as frequencies observed in MIMIC-IV for the same 50 diseases or HPO frequency annotations, and how does their co-occurrence structure compare? Was the GPT-5.5 judge validated against physicians, and do the conclusions hold with a judge from another model family? Which populations do the retrieved literature targets describe, and who reviewed each atlas? Figure 1 mentions one reviewed atlas per disease, but the text does not describe the review. How are severity-conditioned probabilities obtained when sources rarely report frequencies by stratum? What share of patient utterances comes from interaction-local completions outside the ledger (five of thirteen turns in the case study), and are those completions counted in the metrics? Do the main results hold across sampling seeds, and can you report confidence intervals for Table 1 and for the differences between Sol-Qwen gaps? Finally, JD Zhiyi is neither cited nor described. Can the atlas be rebuilt without access to that commercial system?

Reviewer_VgPYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PhenoPatient, a framework generating de novo interactive virtual patients with real world phenotype distributions. PhenoPatient contains disease knowledge construction (M1), patient generation, validation, and freezing (M2), and fact-grounded interaction (M3). PhenoPatient is evaluated against 5 MIMIC-IV record-grounded simulator and one de novo baseline (DemoDx-only) across 50 diseases. The direct evaluation towards the synthetic data is on a comprehensive rubric with one downstream task evaluation (disease accuracy). The results show improvements in several diversity metrics.

Critical strengths
  1. Generating diverse virtual patients through population-level distribution is well motivated. This provides a useful perspective on synthetic data generation, especially in clinical and medical field.

  2. The empirical evaluation is relatively comprehensive. The rubric contains different aspects to evaluate the generated patients, and the evaluation comparison between human experts and LLMs shows useful evidences regarding the robustness of PhenoPatient.

Critical weaknesses
  1. The methodological novelty appears somewhat limit for ICLR. Most of PhenoPatient's main components are based on existing methods. Even though their combination is empirically useful, the paper does not explicitly say which aspects of this integration give a generalizable methodological contribution. Moreover, the experimental comparison is not enough to show the advantages of PhenoPatient over other synthetic data generation methods. The only de novo baseline is DemoDx-only, which is a simplified method baseline constructed by the authors. Although the authors discuss Patient-Zero and state that their code is not available, the lack of stronger de novo generation baselines remains a critical limitation. For example, the authors did not discuss whether LLM-based conditional generation or other probabilistic generation method could achieve comparable performances across the metrics. Since the phenotypes can be combined in different ways, evaluating only 20 generated patients per disease might not be enough to estimate the phenotype distribution and within disease diversity. A sample size sensitivity analysis should be done to evaluate PhenoPatient further. Also, the ablation studies are not enough to isolate the contribution of each component. For example, the removing the immutable ledger or severity conditioning should be other important things to test on.

  2. My other major concern is the validity of the evaluation rubric the authors proposed. For example, Appendix B shows the details of the rubric, but the authors do not clearly state whether these rubrics are adapted from previously validated clinical evaluation or the authors develop these rubrics for this study only. The authors did not compare LLM judge vs. human, and the low inter-rater agreement among human experts further limits the confidence in the reliability of the assessments. Also, one major tool the authors used in M1 is called JD zhiyi, but the paper does not provide a formal reference or sufficient details of this tool, which limits the reproducibility. In addition, the interpretation of the downstream task evaluation in Table. 2 is lack of justification. DemoDx-only achieves higher accuracy than PhenoPatient across all the LLM-based models for both broad category and disease specific diagnosis. The main gain reported in Table. 2 is only a metrics ΔS−Q\Delta_{S-Q}. A larger pairwise gap on accuracy does not necessarily gives better generation quality or even more reliable clinical fidelity. And the justification to choose GPT-5.6 Sol and Qwen is not clear.

Questions for authors
  1. Given that DemoDx-only is the only de novo baseline, could the authors explain why other conditional synthetic data generation methods such as literature-grounded LLM generation or probabilistic generation methods were not considered? Could the authors provide comparisons with those?
  2. Could the authors further clarify the contributions of the individual components used in this paper? For example, how much improvements comes from severity-conditioned sampling compared to disease-conditioned sampling? Also, could the authors isolate the contribution of immutable ledger and other important components?
  3. Were the evaluation rubrics from the Appendix adapted from existing literature of developed specifically for this paper? Have these rubrics been validated by the human clinical experts? Could the authors report the agreement or correlation between the LLM and human experts?
  4. Could the authors provide a sample-size sensitivity analysis on the number of generated patients per disease (i.e. 10, 20, 30, 40, 50)
  5. What motivates using the accuracy gap as the main metrics in Table 2 to show better quality of generated patients? And why selecting particular GPT 5.6-Sol and Qwen3.8-27B? What evidence supports that a larger gap shows a more reliable or more meaningful evaluation?
  6. Could the authors provide a formal reference and additional implementation details for JD Zhiyi? Since the authors mentioned in the paper that the experiments are conducted in Chinese, have the authors considered the potential influence of language on the diagnostic accuracy gaps in Table. 2?
Reviewer_VYYeOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PhenoPatient, a framework for generating clinically plausible and diverse virtual patients without relying on individual real-world medical records. It combines literature-derived, severity-conditioned phenotype distributions, probabilistic sampling, cross-modal consistency correction, and an immutable, rule-checked patient ledger to balance individual-level coherence, population-level distribution alignment, and within-disease diversity. In the experiment, methods are evaluated across 50 diseases, each with 20 simulated cases, and PhenoPatient achieves best results in more than half of the metrics. Blinded physician evaluations and model-backbone experiments provide further evidence of its potential for interactive clinical AI evaluation.

Critical strengths
  1. The paper is well-motivated, addressing a meaningful limitation of virtual-patient simulation: record-grounded systems constrain coverage, while unconstrained de novo generation can produce inconsistent or stereotyped cohorts.
  2. The methodology is reasonable. By combining literature-derived, severity-conditioned phenotype sampling, cross-modal correction, and an immutable patient ledger, PhenoPatient has the potential to provide a coherent framework for jointly targeting individual-level plausibility, population-level agreement, and within-disease diversity.
Critical weaknesses
  1. PhenoPatient aims to generate clinically coherent and diverse patients that can be aligned with real-world phenotype distributions. However, the consistency-correction and rejection/resampling pipeline can alter the initial sampling distribution, which might create bias against the reference distribution. Moreover, the paper does not establish that the final accepted cohort preserves the intended phenotype frequencies, since the ground truth is somewhat ambiguous. Similarly, an immutable ledger ensures state stability but does not guarantee that every interaction-level completion is fully supported by explicit patient facts. These gaps weaken the central claim that the framework simultaneously achieves individual-level coherence and population-level alignment.
  2. The evaluation does not provide sufficient evidence for the full scope of the paper's claims. The reported improvements rely substantially on proxy metrics, including LLM-based plausibility judgments, phenotype-frequency agreement, coverage, and diversity. On the one hand, these measures do not by themselves establish real-world clinical validity or improved downstream clinical reasoning. On the other hand, with only 20 samples per disease, the statistical significance of the metrics might be weak. Moreover, Table 2 actually shows inferior performance of PhenoPatient compared to DemoDx-only regarding diagnostic accuracy, where greater diagnostic separation between doctor models also does not establish greater clinical validity. The paper also does not directly demonstrate that training on the generated patients improves performance on independent clinical tasks. These concerns leave the effectiveness and robustness of the proposed method uncertain.
Questions for authors
  1. Can the authors quantify how phenotype frequencies change from initial sampling to correction and final acceptance, and report whether the accepted cohorts remain aligned with the intended targets? In particular, how does the non-convergence and resampling process affect the reported population-level agreement?
  2. The evaluation relies on interaction-visible phenotype frequencies, LLM-based plausibility judgments, and coverage and diversity metrics. To what extent do these measures establish that the generated patients reflect real-world clinical distributions and clinically meaningful disease trajectories, rather than only satisfying the LLM-judge’s rubric? How do the authors evaluate the clinical validity of the proposed framework?
  3. The main evaluation includes only 20 completed cases per disease, while 532 of 1,532 initially instantiated candidates failed to converge within the permitted repair rounds. Can the authors show whether these samples are enough for augmenting real-world clinical applications, and what are generation cost and latency of the entire process?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.