- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes PhenoPatient, a method for generating virtual patients without real records. An LLM proposes candidate phenotypes in five modalities for each disease, and a commercial retrieval engine (JD Zhiyi) adds missing candidates and literature-derived prevalence and frequency evidence, which together form a severity-conditioned phenotype atlas.
Critical strengths
The comparison is carefully controlled. All seven simulators share the same disease allocation and case slots, the same roles, language and interaction budget, and the doctor-model and patient-model experiments reuse identical frozen panels, so differences between rows are not explained by different patient samples. The paper also reports what other papers often omit, including the 35% non-convergence rate of candidate patients, operational definitions for every metric, a fully traced example case, and Holm-adjusted paired tests for the physician study. This matters for my assessment because the infrastructure would support a sound comparison if the reference targets were independent.
A second strength is the framing. Separating individual-level plausibility, population-level agreement and within-disease diversity makes explicit a trade-off that patient simulators usually leave implicit: typical cases are coherent but uniform, while broad sampling creates contradictions. Record-free generation also addresses a real constraint on sharing clinical data.
Critical weaknesses
My main concern is that the population-level and diversity results, which carry the paper's central claim of alignment with real-world phenotype distributions, are largely circular. The prevalence, phenotype-frequency, course-length and numeric targets used for scoring come from the same literature retrieval that parameterized PhenoPatient's atlas, and the coverage denominators are candidate sets derived from the same evidence. PhenoPatient therefore samples from the distributions it is scored against, while the record-grounded simulators never saw them. Real MIMIC-IV patients score far below the synthetic cohort on these measures, which shows that the metrics capture conformity to the target specification more than realism. Differences between the literature populations and US inpatients, and the Chinese-language rendering of English records, would penalize the record-grounded simulators further. The ablations point the same way, since Direct Values, which stops sampling from literature ranges, loses 11.61 points of IQR matching. The metrics also test marginal rates only. Phenotypes are drawn independently given disease and severity, so their co-occurrence is unconstrained, and the case study's patient with mild disease activity carries 51 positive findings. An independent reference is needed, such as phenotype frequencies observed in MIMIC-IV for the same diseases, held-out literature, or HPO frequency annotations, together with a check of joint co-occurrence.
The evidence for individual plausibility has its own problem: the judges may prefer generated cases, and the diagnosis results suggest those cases are easier than real ones. GPT-5.5 plays the patient and the physician and also scores all four judged plausibility metrics, with no validation of the judge against clinicians. The LLM-only baseline, which invents a whole case from age, sex and diagnosis, receives the highest symptom and sign plausibility of all seven simulators and higher course plausibility than every simulator built from real records, which suggests the judge rewards textbook typicality. The physician study shows the same pattern and adds a reliability problem. Krippendorff's alpha lies between 0.03 and 0.10, 65% of all ratings are 4, and blinding is hard to maintain when record-derived cases visibly differ from generated ones. In the doctor experiment, the three strongest models reach 98 to 99% broad-category accuracy on PhenoPatient cases, against 71 to 81% on the record-grounded simulators. The largest Sol-Qwen gap therefore comes from the weaker model while the stronger one is at ceiling, and the differences between gaps (25.2 against 22.0, and 35.0 against 31.4) come from single runs with no uncertainty estimates.
Questions for authors
Some clarifications could change my assessment. How do PhenoPatient's revealed phenotype frequencies compare with an independent reference, such as frequencies observed in MIMIC-IV for the same 50 diseases or HPO frequency annotations, and how does their co-occurrence structure compare? Was the GPT-5.5 judge validated against physicians, and do the conclusions hold with a judge from another model family? Which populations do the retrieved literature targets describe, and who reviewed each atlas? Figure 1 mentions one reviewed atlas per disease, but the text does not describe the review. How are severity-conditioned probabilities obtained when sources rarely report frequencies by stratum? What share of patient utterances comes from interaction-local completions outside the ledger (five of thirteen turns in the case study), and are those completions counted in the metrics? Do the main results hold across sampling seeds, and can you report confidence intervals for Table 1 and for the differences between Sol-Qwen gaps? Finally, JD Zhiyi is neither cited nor described. Can the atlas be rebuilt without access to that commercial system?