- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper presents EMRGuide-Zero, an adversarial dual-LLM framework designed to synthesize clinical electronic medical records (EMRs) without relying on manually authored guidelines. The framework decouples synthesis guidance into two automatically induced categories: 1) Content Guidelines: Operationalized via executable regex pattern-matching detectors () that capture observable clinical findings. Their occurrence frequencies and pairwise co-occurrences () are programmatically measured across the real training corpus and sampled via diagnosis-specific Chow–Liu dependence trees to plan record content. 2) Documentation Guidelines: Formulated as condition-target pairs specifying how clinical findings should be documented given a specific context . These are validated against the real corpus using an empirical satisfaction threshold () and applied via targeted conditional refinement on draft records. An Inducer LLM contrasts batches of real and synthetic records to iteratively identify residual documentation gaps, while a Generator LLM produces records conditioned on accepted guidelines. The authors evaluate EMRGuide-Zero across two clinical datasets (EMR-302, MIMIC-47) using three LLM backbones (Qwen3.5-9B, Gemma-4-12B-it, Llama-3.1-8B-Instruct), reporting improvements in distribution matching, semantic coverage, downstream diagnosis classification, privacy preservation, and cross-model guideline transferability.
Critical strengths
- Framing guideline construction as an iterative, adversarial contrastive learning loop effectively circumvents the high cost, incomplete coverage, and rigidity of manual guideline authoring.
- The distinction between global content planning (via empirical co-occurrence distributions and Chow–Liu trees) and local documentation refinement (via conditional checks) is well-motivated and structurally sound.
- The experimental evaluation is extensive across automated dimensions: testing on two diverse datasets, three modern open-weight LLMs, four competitive baselines (Direct, MedSyn, ClinGen, LLM-CARe), downstream utility via fine-tuned classifier transfer, cross-backbone guideline portability, and membership inference attacks.
- The framework achieves high semantic alignment with demonstrably lower n-gram overlap and membership leakage compared to few-shot retrieval-augmented baselines.
Critical weaknesses
-
Absence of Clinical Expert Evaluation. Despite synthesizing complex clinical narratives, the evaluation relies entirely on automated metrics (MAE of entity frequencies, joint TVD, MMD², embedding cosine distance) and a simplified 4-choice classification probe. There is no qualitative or quantitative evaluation by medical doctors or certified clinical annotators. Automated distributional metrics cannot detect subtle clinical contradictions, medical hallucinations, pharmacological contraindications, or nonsensical temporal progressions in hospital courses. For high-stakes medical text synthesis, the lack of expert verification is a major limitation.
-
Unverified Correctness of Induced Guidelines. The framework assumes that candidate guidelines induced by the LLM are clinically valid if they pass statistical screening ( recurrence and ). However, statistical correlation in real records does not guarantee clinical soundness. Without clinical audit of the induced guideline bank, the Inducer LLM may capture spurious correlations, non-standard documentation habits, or hallucinated clinical rationales. Furthermore, the content detectors () are LLM-generated regular expressions. Regex pattern matching in free-text clinical notes is notoriously brittle and vulnerable to negation, uncertainty, hedging (e.g., "patient denies fever" vs. "patient has fever"), and linguistic variations.
-
Missing Component-Level Ablations. While Table 2 provides a step-wise ablation comparing
Direct,+Content, and+Documentation, it only evaluates the broad two-stage generation pipeline. Key algorithmic components of the framework remain unablated:- The necessity and impact of the Chow–Liu dependency tree vs. simple independent content sampling or full joint estimation.
- Sensitivity to the empirical validation threshold ().
- The specific contribution of batch contrast vs. pairwise contrast in the Inducer.
- The utility of the programmatic replay verification vs. direct unvalidated LLM guideline generation.
-
Limited Downstream Task Scope. First, the testbed is limited. Downstream utility is evaluated solely on a synthetic-train-real-test 4-choice diagnosis prediction probe using a small 0.8B parameter model, restricted only to Chief Complaint and History of Present Illness fields. Additionally, the evaluation tasks are limited. The synthetic records are not evaluated on more realistic, complex clinical NLP tasks such as ICD-10 multi-label code assignment, mortality/readmission prediction, or clinical information extraction.
Questions for authors
- Please perform manual or clinical review on a random sample of the induced content and documentation guidelines. Report the proportion of induced rules that represent medically sound principles versus spurious reporting artifacts.
- How do the authors verify that the synthesized notes exhibit clinical authenticity rather than just statistical mimicry?
- Did any medical practitioners assess whether these synthetic records are distinguishable from genuine clinical documentation (e.g., an authenticity Turing test)?
- How does the framework ensure that generated notes do not contain medically unauthentic statements (e.g., mutually contradictory diagnoses or hallucinated clinical pathways) that automated fidelity metrics are blind to?
- How do the LLM-generated bounded regular expressions handle clinical negation, historical conditions, and family history? What is their empirical precision and recall against human-annotated or gold-standard entity spans?
- How sensitive is the documentation guideline set to the choice of ? Does lowering or raising lead to guideline under-fitting or noisy over-generation?
- What is the total inference overhead (token consumption, wall-clock time) required to complete the 100 induction rounds across 47/302 diagnosis groups compared to the one-time manual guideline construction?
6 comments
Response to Reviewer 5sop (2/3)
- Further Ablation Study: Thank you for these suggestions. We conducted the following ablations with Qwen3.5-9B. Independent Sampling samples content guidelines from their marginal frequencies without the Chow–Liu tree. τ = 0.7 and τ = 0.9 vary the validation threshold of documentation guidelines. Pairwise splits each batch contrast into parallel contrasts between one real and one synthetic record, so that the Inducer examines the same number of records per round.
| Dataset | Variant | Entity freq. ↓ | Entity cooc. ↓ | Semantic sim. ↓ | Semantic cov. ↓ | Diagnosis pred. ↑ |
|---|---|---|---|---|---|---|
| EMR-302 | Independent Sampling | 0.166 | 0.271 | 0.083 | 0.205 | 0.842 |
| EMR-302 | τ = 0.7 | 0.166 | 0.269 | 0.081 | 0.204 | 0.850 |
| EMR-302 | τ = 0.9 | 0.160 | 0.265 | 0.083 | 0.209 | 0.848 |
| EMR-302 | Pairwise | 0.161 | 0.264 | 0.079 | 0.203 | 0.849 |
| EMR-302 | EMRGuide-Zero (ours) | 0.159 | 0.264 | 0.076 | 0.201 | 0.854 |
| MIMIC-47 | Independent Sampling | 0.197 | 0.318 | 0.149 | 0.284 | 0.832 |
| MIMIC-47 | τ = 0.7 | 0.186 | 0.304 | 0.151 | 0.287 | 0.843 |
| MIMIC-47 | τ = 0.9 | 0.187 | 0.306 | 0.146 | 0.285 | 0.838 |
| MIMIC-47 | Pairwise | 0.185 | 0.303 | 0.150 | 0.282 | 0.848 |
| MIMIC-47 | EMRGuide-Zero (ours) | 0.184 | 0.303 | 0.142 | 0.281 | 0.844 |
First, Independent Sampling degrades all metrics on both datasets, most notably entity co-occurrence, showing that dependencies among content guidelines matter for planning realistic records. Estimating the full joint distribution is infeasible, since the number of configurations grows exponentially with the number of guidelines, whereas the Chow–Liu tree is a tree-structured approximation that can be estimated from pairwise statistics. Second, both a lower and a higher threshold slightly reduce performance: a lower threshold admits requirements that real records do not consistently follow, whereas a higher one rejects useful requirements. The variation remains small, indicating that the method is not sensitive to the exact choice of τ. Third, Pairwise contrast performs comparably on entity metrics but is weaker in semantic similarity, since any two records differ in many incidental ways, whereas batch contrast exposes differences that recur across records. Finally, replaying detectors over the training records is what provides the frequencies and co-occurrences for content planning, so its contribution is reflected in the +Content step of Table 2.
Response to Reviewer 5sop (3/3)
- Extended Downstream Evaluation: Thank you for this suggestion. Using the Qwen3.5-9B-generated corpora, we extend downstream evaluation with a second fine-tuned model, Llama-3.2-1B-Instruct, and three additional tasks: examination and treatment selection following LLM-CARe, and open-ended diagnosis prediction, where the model generates the diagnosis name without options. Since each synthetic record is generated for a given diagnosis group and does not carry outcome labels or span annotations, we focus on tasks whose labels can be derived from the records themselves. EMRGuide-Zero achieves the best accuracy in 14 of the 16 task–model–dataset settings, including all settings of open-ended diagnosis prediction, indicating that its utility is not tied to a particular task or downstream model.
| Dataset | Downstream Model | Method | Diagnosis Selection ↑ | Examination Selection ↑ | Treatment Selection ↑ | Open-Ended Diagnosis ↑ |
|---|---|---|---|---|---|---|
| EMR-302 | Qwen3.5-0.8B | Direct | 0.821 | 0.605 | 0.652 | 0.323 |
| EMR-302 | Qwen3.5-0.8B | MedSyn | 0.842 | 0.749 | 0.724 | 0.374 |
| EMR-302 | Qwen3.5-0.8B | ClinGen | 0.817 | 0.756 | 0.743 | 0.365 |
| EMR-302 | Qwen3.5-0.8B | LLM-CARe | 0.826 | 0.789 | 0.778 | 0.352 |
| EMR-302 | Qwen3.5-0.8B | EMRGuide-Zero (ours) | 0.854 | 0.812 | 0.811 | 0.387 |
| EMR-302 | Llama-3.2-1B-Instruct | Direct | 0.805 | 0.597 | 0.624 | 0.317 |
| EMR-302 | Llama-3.2-1B-Instruct | MedSyn | 0.830 | 0.739 | 0.701 | 0.368 |
| EMR-302 | Llama-3.2-1B-Instruct | ClinGen | 0.810 | 0.730 | 0.725 | 0.358 |
| EMR-302 | Llama-3.2-1B-Instruct | LLM-CARe | 0.814 | 0.769 | 0.751 | 0.344 |
| EMR-302 | Llama-3.2-1B-Instruct | EMRGuide-Zero (ours) | 0.851 | 0.786 | 0.794 | 0.387 |
| MIMIC-47 | Qwen3.5-0.8B | Direct | 0.807 | 0.888 | 0.889 | 0.348 |
| MIMIC-47 | Qwen3.5-0.8B | MedSyn | 0.821 | 0.929 | 0.946 | 0.382 |
| MIMIC-47 | Qwen3.5-0.8B | ClinGen | 0.824 | 0.934 | 0.948 | 0.399 |
| MIMIC-47 | Qwen3.5-0.8B | LLM-CARe | 0.785 | 0.939 | 0.961 | 0.393 |
| MIMIC-47 | Qwen3.5-0.8B | EMRGuide-Zero (ours) | 0.844 | 0.949 | 0.942 | 0.401 |
| MIMIC-47 | Llama-3.2-1B-Instruct | Direct | 0.786 | 0.885 | 0.907 | 0.363 |
| MIMIC-47 | Llama-3.2-1B-Instruct | MedSyn | 0.819 | 0.911 | 0.946 | 0.396 |
| MIMIC-47 | Llama-3.2-1B-Instruct | ClinGen | 0.818 | 0.933 | 0.949 | 0.407 |
| MIMIC-47 | Llama-3.2-1B-Instruct | LLM-CARe | 0.816 | 0.943 | 0.948 | 0.390 |
| MIMIC-47 | Llama-3.2-1B-Instruct | EMRGuide-Zero (ours) | 0.842 | 0.928 | 0.962 | 0.420 |
- Computational Cost: Thank you for this question. With Qwen3.5-9B served by vLLM on a single RTX 4090 GPU, guideline induction consumes 313.0M tokens and 83.0 GPU hours on EMR-302, and 94.5M tokens and 20.1 GPU hours on MIMIC-47, corresponding to about 0.27 and 0.43 GPU hours per diagnosis group. Induction is performed once per corpus, after which the bank is frozen and can be reused by other generators without repeating induction (Figure 4). Diagnosis groups are induced independently and can therefore be processed in parallel when more resources are available. Existing methods such as LLM-CARe use a small set of manually summarized guidelines shared across diagnoses. Obtaining fine-grained, diagnosis-specific guidelines comparable to ours manually would require experts to author them for each diagnosis group, which involves more effort than these shared guidelines, whereas our induction produces them automatically.
Response to Reviewer 5sop (1/3)
EMRGuide-Zero has the fewest medical errors, comparable consistency and completeness to MedSyn, and is more often judged real (70% vs. 55%), approaching real records (80%). The lower completeness of real records mainly arises because some of their fields are left empty or filled with placeholders. The clinicians also rated 20 sampled guidelines of each type on the same scale. Documentation guidelines score 9.25, and content guidelines score 8.58 with no severe problems. The minor problems in content guidelines mostly concern findings or medications that are not typical of the diagnosis and likely stem from comorbidities. Since content guidelines reproduce how often such content appears in real records rather than requiring it in every record, they remain consistent with the source data.