acceptodds
Under review as a conference paper at ICLR 2027

EMRGuide-Zero: Synthesizing Electronic Medical Records via Dual-LLM Adversarial Guideline Induction from Samples

Abstract

Electronic medical records (EMRs) are a valuable source for clinical research, yet privacy restrictions make them difficult to access and share. Synthetic EMRs offer a practical alternative, and recent methods steer large language models (LLMs) with manually summarized synthesis guidelines, often alongside real examples. Because the generator follows what these guidelines specify, fidelity hinges on them, yet manual summaries leave many context-specific requirements unstated and omit the conditions under which others apply. To address these limitations, we introduce EMRGuide-Zero, a dual-LLM adversarial framework that induces fine-grained synthesis guidelines from samples with zero manual guideline authoring. At each round, an Inducer LLM contrasts real and synthetic records to propose candidate guidelines, which are tested on broader samples to filter out local coincidences. Each guideline also specifies where it applies, allowing a Generator LLM to impose it only on the cases it concerns. The validated guidelines are incorporated into subsequent generation rounds, where new contrasts expose requirements that earlier rounds missed. Experiments across multiple EMR corpora and LLM backbones show that EMRGuide-Zero discovers fine-grained, clinically meaningful guidelines and improves fidelity and downstream utility while keeping privacy risk close to generation without direct use of real record text. These results establish guideline induction from samples as an effective alternative to manual authoring for synthesizing information-rich EMRs across clinical contexts.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Synthetic Data Generation, Large Language Model, Electronic Medical Record

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5sopOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents EMRGuide-Zero, an adversarial dual-LLM framework designed to synthesize clinical electronic medical records (EMRs) without relying on manually authored guidelines. The framework decouples synthesis guidance into two automatically induced categories: 1) Content Guidelines: Operationalized via executable regex pattern-matching detectors (γj\gamma_j) that capture observable clinical findings. Their occurrence frequencies and pairwise co-occurrences (πjℓ\pi_{j\ell}) are programmatically measured across the real training corpus and sampled via diagnosis-specific Chow–Liu dependence trees to plan record content. 2) Documentation Guidelines: Formulated as condition-target pairs (Ck,Tk)(C_k, T_k) specifying how clinical findings should be documented given a specific context CC. These are validated against the real corpus using an empirical satisfaction threshold (τ=0.8\tau = 0.8) and applied via targeted conditional refinement on draft records. An Inducer LLM contrasts batches of real and synthetic records to iteratively identify residual documentation gaps, while a Generator LLM produces records conditioned on accepted guidelines. The authors evaluate EMRGuide-Zero across two clinical datasets (EMR-302, MIMIC-47) using three LLM backbones (Qwen3.5-9B, Gemma-4-12B-it, Llama-3.1-8B-Instruct), reporting improvements in distribution matching, semantic coverage, downstream diagnosis classification, privacy preservation, and cross-model guideline transferability.

Critical strengths
  1. Framing guideline construction as an iterative, adversarial contrastive learning loop effectively circumvents the high cost, incomplete coverage, and rigidity of manual guideline authoring.
  2. The distinction between global content planning (via empirical co-occurrence distributions and Chow–Liu trees) and local documentation refinement (via conditional (Ck,Tk)(C_k, T_k) checks) is well-motivated and structurally sound.
  3. The experimental evaluation is extensive across automated dimensions: testing on two diverse datasets, three modern open-weight LLMs, four competitive baselines (Direct, MedSyn, ClinGen, LLM-CARe), downstream utility via fine-tuned classifier transfer, cross-backbone guideline portability, and membership inference attacks.
  4. The framework achieves high semantic alignment with demonstrably lower n-gram overlap and membership leakage compared to few-shot retrieval-augmented baselines.
Critical weaknesses
  1. Absence of Clinical Expert Evaluation. Despite synthesizing complex clinical narratives, the evaluation relies entirely on automated metrics (MAE of entity frequencies, joint TVD, MMD², embedding cosine distance) and a simplified 4-choice classification probe. There is no qualitative or quantitative evaluation by medical doctors or certified clinical annotators. Automated distributional metrics cannot detect subtle clinical contradictions, medical hallucinations, pharmacological contraindications, or nonsensical temporal progressions in hospital courses. For high-stakes medical text synthesis, the lack of expert verification is a major limitation.

  2. Unverified Correctness of Induced Guidelines. The framework assumes that candidate guidelines induced by the LLM are clinically valid if they pass statistical screening (πjℓ\pi_{j\ell} recurrence and ρkreal>0.8\rho_k^{\text{real}} > 0.8). However, statistical correlation in real records does not guarantee clinical soundness. Without clinical audit of the induced guideline bank, the Inducer LLM may capture spurious correlations, non-standard documentation habits, or hallucinated clinical rationales. Furthermore, the content detectors (γj\gamma_j) are LLM-generated regular expressions. Regex pattern matching in free-text clinical notes is notoriously brittle and vulnerable to negation, uncertainty, hedging (e.g., "patient denies fever" vs. "patient has fever"), and linguistic variations.

  3. Missing Component-Level Ablations. While Table 2 provides a step-wise ablation comparing Direct, +Content, and +Documentation, it only evaluates the broad two-stage generation pipeline. Key algorithmic components of the framework remain unablated:

    • The necessity and impact of the Chow–Liu dependency tree vs. simple independent content sampling or full joint estimation.
    • Sensitivity to the empirical validation threshold (τ=0.8\tau = 0.8).
    • The specific contribution of batch contrast vs. pairwise contrast in the Inducer.
    • The utility of the programmatic replay verification vs. direct unvalidated LLM guideline generation.
  4. Limited Downstream Task Scope. First, the testbed is limited. Downstream utility is evaluated solely on a synthetic-train-real-test 4-choice diagnosis prediction probe using a small 0.8B parameter model, restricted only to Chief Complaint and History of Present Illness fields. Additionally, the evaluation tasks are limited. The synthetic records are not evaluated on more realistic, complex clinical NLP tasks such as ICD-10 multi-label code assignment, mortality/readmission prediction, or clinical information extraction.

Questions for authors
  1. Please perform manual or clinical review on a random sample of the induced content and documentation guidelines. Report the proportion of induced rules that represent medically sound principles versus spurious reporting artifacts.
  2. How do the authors verify that the synthesized notes exhibit clinical authenticity rather than just statistical mimicry?
  3. Did any medical practitioners assess whether these synthetic records are distinguishable from genuine clinical documentation (e.g., an authenticity Turing test)?
  4. How does the framework ensure that generated notes do not contain medically unauthentic statements (e.g., mutually contradictory diagnoses or hallucinated clinical pathways) that automated fidelity metrics are blind to?
  5. How do the LLM-generated bounded regular expressions handle clinical negation, historical conditions, and family history? What is their empirical precision and recall against human-annotated or gold-standard entity spans?
  6. How sensitive is the documentation guideline set Gdoc\mathcal{G}_{\text{doc}} to the choice of τ=0.8\tau = 0.8? Does lowering or raising τ\tau lead to guideline under-fitting or noisy over-generation?
  7. What is the total inference overhead (token consumption, wall-clock time) required to complete the 100 induction rounds across 47/302 diagnosis groups compared to the one-time manual guideline construction?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 5sop (1/3)

  1. Clinical Assessment by Physicians: Thank you for this suggestion. Three licensed clinicians independently assessed 20 records each from the real corpus, MedSyn (one of the strongest baselines overall in Table 1), and EMRGuide-Zero with Qwen3.5-9B on EMR-302. Without knowing the source, they rated each record for medical errors, internal contradictions, and incompleteness due to missing key information, and judged whether it was real or synthetic. Each rating uses three levels, which we score as 10 (no problem), 5 (minor problem that does not seriously affect clinical use), and 0 (severe problem). Inter-annotator agreement is substantial, with Krippendorff's α of 0.75–0.80 for the three quality ratings and Fleiss' κ of 0.60 for authenticity.
SourceMedical Correctness ↑Consistency ↑Completeness ↑Judged Real ↑
Real8.589.425.9280%
MedSyn8.508.507.8355%
EMRGuide-Zero (ours)9.258.177.9270%

EMRGuide-Zero has the fewest medical errors, comparable consistency and completeness to MedSyn, and is more often judged real (70% vs. 55%), approaching real records (80%). The lower completeness of real records mainly arises because some of their fields are left empty or filled with placeholders. The clinicians also rated 20 sampled guidelines of each type on the same scale. Documentation guidelines score 9.25, and content guidelines score 8.58 with no severe problems. The minor problems in content guidelines mostly concern findings or medications that are not typical of the diagnosis and likely stem from comorbidities. Since content guidelines reproduce how often such content appears in real records rather than requiring it in every record, they remain consistent with the source data.

  1. Details of Content Detectors: Thank you for this question. Past and family history are usually documented in separate fields, and matching each detector within its field distinguishes most of these cases. Negation is harder to handle with fixed patterns, as keywords can be mismatched and the many ways of expressing negation are difficult to enumerate. Instead of designing such rules manually, we let the Inducer propose negated forms as separate content guidelines when contrasts reveal them, in line with our goal of avoiding manual authoring. Their frequencies and co-occurrences are measured like those of other guidelines, and when both a finding and its negated form are sampled for a record, the Generator writes only the negated form. We analyzed the content guidelines induced by Qwen3.5-9B on EMR-302 against the entity annotations of the dataset. First, the induced guidelines do capture negation. In the present-illness field, where negative findings are most commonly documented, 16.7% of content guidelines are negated forms. To check negation in the synthetic records beyond our own detectors, we take the annotated symptoms and, for evaluation only, treat a mention as negated when a negation cue precedes it within the same clause. Across all annotated symptoms, including those without a negated guideline, EMRGuide-Zero has the lowest error in the per-symptom rate of negated mentions among all methods (0.124 vs. 0.126–0.182 for the baselines), and the closest distribution of affirmed and negated mentions to held-out real records (total variation distance 0.186 vs. 0.207–0.305). Second, regarding precision and recall, 84.8% of content guidelines involve an annotated clinical entity. Of these, 49.7% match the entity exactly, while others add information about it, such as its negation or the result of an examination. The remaining guidelines capture documented content beyond entities, such as follow-up and dietary advice. Although induction relies on no external vocabulary, the guidelines cover 60.5% of the entities that occur more than 50 times in the training records. These results suggest that the induced detectors capture both negation and a substantial portion of clinical content automatically, while also extending to documented information beyond predefined entities.
AuthorsOpenReview ↗

Response to Reviewer 5sop (2/3)

  1. Further Ablation Study: Thank you for these suggestions. We conducted the following ablations with Qwen3.5-9B. Independent Sampling samples content guidelines from their marginal frequencies without the Chow–Liu tree. τ = 0.7 and τ = 0.9 vary the validation threshold of documentation guidelines. Pairwise splits each batch contrast into parallel contrasts between one real and one synthetic record, so that the Inducer examines the same number of records per round.
DatasetVariantEntity freq. ↓Entity cooc. ↓Semantic sim. ↓Semantic cov. ↓Diagnosis pred. ↑
EMR-302Independent Sampling0.1660.2710.0830.2050.842
EMR-302τ = 0.70.1660.2690.0810.2040.850
EMR-302τ = 0.90.1600.2650.0830.2090.848
EMR-302Pairwise0.1610.2640.0790.2030.849
EMR-302EMRGuide-Zero (ours)0.1590.2640.0760.2010.854
MIMIC-47Independent Sampling0.1970.3180.1490.2840.832
MIMIC-47τ = 0.70.1860.3040.1510.2870.843
MIMIC-47τ = 0.90.1870.3060.1460.2850.838
MIMIC-47Pairwise0.1850.3030.1500.2820.848
MIMIC-47EMRGuide-Zero (ours)0.1840.3030.1420.2810.844

First, Independent Sampling degrades all metrics on both datasets, most notably entity co-occurrence, showing that dependencies among content guidelines matter for planning realistic records. Estimating the full joint distribution is infeasible, since the number of configurations grows exponentially with the number of guidelines, whereas the Chow–Liu tree is a tree-structured approximation that can be estimated from pairwise statistics. Second, both a lower and a higher threshold slightly reduce performance: a lower threshold admits requirements that real records do not consistently follow, whereas a higher one rejects useful requirements. The variation remains small, indicating that the method is not sensitive to the exact choice of τ. Third, Pairwise contrast performs comparably on entity metrics but is weaker in semantic similarity, since any two records differ in many incidental ways, whereas batch contrast exposes differences that recur across records. Finally, replaying detectors over the training records is what provides the frequencies and co-occurrences for content planning, so its contribution is reflected in the +Content step of Table 2.

AuthorsOpenReview ↗

Response to Reviewer 5sop (3/3)

  1. Extended Downstream Evaluation: Thank you for this suggestion. Using the Qwen3.5-9B-generated corpora, we extend downstream evaluation with a second fine-tuned model, Llama-3.2-1B-Instruct, and three additional tasks: examination and treatment selection following LLM-CARe, and open-ended diagnosis prediction, where the model generates the diagnosis name without options. Since each synthetic record is generated for a given diagnosis group and does not carry outcome labels or span annotations, we focus on tasks whose labels can be derived from the records themselves. EMRGuide-Zero achieves the best accuracy in 14 of the 16 task–model–dataset settings, including all settings of open-ended diagnosis prediction, indicating that its utility is not tied to a particular task or downstream model.
DatasetDownstream ModelMethodDiagnosis Selection ↑Examination Selection ↑Treatment Selection ↑Open-Ended Diagnosis ↑
EMR-302Qwen3.5-0.8BDirect0.8210.6050.6520.323
EMR-302Qwen3.5-0.8BMedSyn0.8420.7490.7240.374
EMR-302Qwen3.5-0.8BClinGen0.8170.7560.7430.365
EMR-302Qwen3.5-0.8BLLM-CARe0.8260.7890.7780.352
EMR-302Qwen3.5-0.8BEMRGuide-Zero (ours)0.8540.8120.8110.387
EMR-302Llama-3.2-1B-InstructDirect0.8050.5970.6240.317
EMR-302Llama-3.2-1B-InstructMedSyn0.8300.7390.7010.368
EMR-302Llama-3.2-1B-InstructClinGen0.8100.7300.7250.358
EMR-302Llama-3.2-1B-InstructLLM-CARe0.8140.7690.7510.344
EMR-302Llama-3.2-1B-InstructEMRGuide-Zero (ours)0.8510.7860.7940.387
MIMIC-47Qwen3.5-0.8BDirect0.8070.8880.8890.348
MIMIC-47Qwen3.5-0.8BMedSyn0.8210.9290.9460.382
MIMIC-47Qwen3.5-0.8BClinGen0.8240.9340.9480.399
MIMIC-47Qwen3.5-0.8BLLM-CARe0.7850.9390.9610.393
MIMIC-47Qwen3.5-0.8BEMRGuide-Zero (ours)0.8440.9490.9420.401
MIMIC-47Llama-3.2-1B-InstructDirect0.7860.8850.9070.363
MIMIC-47Llama-3.2-1B-InstructMedSyn0.8190.9110.9460.396
MIMIC-47Llama-3.2-1B-InstructClinGen0.8180.9330.9490.407
MIMIC-47Llama-3.2-1B-InstructLLM-CARe0.8160.9430.9480.390
MIMIC-47Llama-3.2-1B-InstructEMRGuide-Zero (ours)0.8420.9280.9620.420
  1. Computational Cost: Thank you for this question. With Qwen3.5-9B served by vLLM on a single RTX 4090 GPU, guideline induction consumes 313.0M tokens and 83.0 GPU hours on EMR-302, and 94.5M tokens and 20.1 GPU hours on MIMIC-47, corresponding to about 0.27 and 0.43 GPU hours per diagnosis group. Induction is performed once per corpus, after which the bank is frozen and can be reused by other generators without repeating induction (Figure 4). Diagnosis groups are induced independently and can therefore be processed in parallel when more resources are available. Existing methods such as LLM-CARe use a small set of manually summarized guidelines shared across diagnoses. Obtaining fine-grained, diagnosis-specific guidelines comparable to ours manually would require experts to author them for each diagnosis group, which involves more effort than these shared guidelines, whereas our induction produces them automatically.
Reviewer_SCBZOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes EMRGuide-Zero, an iterative dual-LLM framework for synthesizing clinical notes without manually authored synthesis guidelines. An Inducer compares real and synthetic records, proposes candidate content and documentation guidelines, and validates them against broader training samples. A Generator then uses the accumulated guidelines to synthesize new records, and the process repeats to identify remaining discrepancies. Content guidelines are represented by regex-like detectors and used with a Chow–Liu dependency model for corpus-level sampling, while documentation guidelines specify applicability conditions for conditional refinement. Experiments on two datasets (EMR-302 and MIMIC-47) and three LLM backbones show generally improved distributional metrics and the best diagnosis-prediction accuracy across all six dataset/backbone settings compared with the reported baselines.

Critical strengths

Automatic, validated guideline induction is a meaningful advance over fixed or supplied guidelines. Separating what content should occur from when and how a documentation requirement applies is sensible. Contrasting real and synthetic records is a natural way to discover corpus-specific requirements rather than hard-coding them. Separating proposal from corpus-level validation is also a good design choice, because it reduces the chance that incidental details from a few records become synthesis constraints.

Critical weaknesses
  1. The comparisons and ablations do not isolate the claimed contribution, iterative guideline induction. The baselines all use supplied or fixed guidelines. There is no comparison with automatic prompt-optimization or self-refinement methods (e.g., APE, OPRO, TextGrad, DSPy, Self-Refine/Reflexion), or with the feedback-loop approach, e.g., Lee et al. (2026). The ablation covers only content planning and conditional refinement. It lacks controls such as the same pipeline with a guideline bank induced once (non-iteratively) or with hand-specified detectors. Figure 5 shows gains growing with rounds, but this does not separate the effect of iteration from simply having more guidelines or more LLM compute. Moreover, several gains are small (especially diagnosis accuracy and the EMR-302 ablation), and no confidence intervals or significance tests are reported.
  2. The evaluation is narrow and favorable to the method’s setup. Distributional closeness does not guarantee clinical correctness, yet there is no expert evaluation of factuality, readability, or safety. Utility is measured on a single downstream task (diagnosis prediction) with a single fine-tuned model (Qwen3.5-0.8B). Every record is generated conditioned on its diagnosis and induction is diagnosis-local, so every evaluated diagnosis has its own induced guideline bank. This is an interpolation regime, and it is untested for unseen diagnoses or for guideline transfer across corpora (only cross-backbone transfer is shown).
Questions for authors
  1. Figure 5 shows several metrics still improving at round 100. Why is this limit considered sufficient? Could the authors report results with a larger induction budget and clarify the quality–compute trade-off?
  2. How are the applicability conditions (C_k) and targets (T_k) specified and implemented? Are they represented by executable checks, and how are these checks generated and validated?
  3. Does the Inducer receive raw real-record text in its prompts? How do the authors prevent patient-specific information from being encoded in the induced guideline bank, and have they evaluated leakage from the bank itself rather than only from the final synthetic records?
7 comments
AuthorsOpenReview ↗

Response to Reviewer SCBZ (1/3)

  1. Comparison with Prompt Optimization Methods: We sincerely appreciate your recognition of automatic guideline induction and your suggestion to compare with these methods. Among our baselines, LLM-CARe represents feedback-based refinement: it has the LLM critique and revise records against predefined quality dimensions. Following your suggestion, we further compared with OPRO, which iteratively optimizes an instruction, using the distributional similarity between a batch of synthetic records and a validation set of real records as its objective, and with Genetic Prompt, which generates diverse data by recombining attributes. Both use Qwen3.5-9B with the same conditions, output schema, and corpus size as ours.
DatasetMethodEntity Freq. ↓Entity Cooc. ↓Semantic Sim. ↓Semantic Cov. ↓Diagnosis Pred. ↑
EMR-302OPRO0.2910.4610.1880.2460.817
EMR-302Genetic Prompt0.2650.4160.1680.2350.817
EMR-302EMRGuide-Zero (ours)0.1590.2640.0760.2010.854
MIMIC-47OPRO0.2610.4010.2350.3130.821
MIMIC-47Genetic Prompt0.2690.4180.2840.3190.812
MIMIC-47EMRGuide-Zero (ours)0.1840.3030.1420.2810.844

EMRGuide-Zero outperforms both methods on all metrics across both datasets. Beyond general prompt optimization, our design considers two characteristics of EMR synthesis. First, patients with the same diagnosis differ in findings and disease courses, so the corpus must cover varied clinical situations; a single optimized instruction per diagnosis, as in OPRO, tends to concentrate on typical cases, whereas EMRGuide-Zero plans each record's content from measured frequencies and co-occurrences and applies documentation guidelines only where their conditions hold. Second, content in some real records may arise only under particular patient circumstances, so it should be validated before being generalized; recombining attributes, as in Genetic Prompt, may combine content from different patient conditions, whereas EMRGuide-Zero replays content detectors over the full training set and tests documentation guidelines on held-out real records.

  1. Effect of Iterative Induction: Thank you for this suggestion. To separate iteration from the number of rounds and LLM calls, we compare with a Non-Iterative variant that runs the same number of induction rounds with the same contrast size, but contrasts real records with the initial synthetic records in every round instead of those generated under the current bank. EMRGuide-Zero outperforms this variant on all metrics in both datasets. With a fixed synthetic side, repeated contrasts keep exposing the same salient gaps, whereas iteration lets closed gaps give way to finer ones in later rounds.
DatasetMethodEntity Freq. ↓Entity Cooc. ↓Semantic Sim. ↓Semantic Cov. ↓Diagnosis Pred. ↑
EMR-302Non-Iterative0.1660.2840.0830.2070.839
EMR-302EMRGuide-Zero (ours)0.1590.2640.0760.2010.854
MIMIC-47Non-Iterative0.2000.3240.1530.2880.835
MIMIC-47EMRGuide-Zero (ours)0.1840.3030.1420.2810.844

Regarding the round limit, our preliminary experiments showed that gains beyond 100 rounds were small, so we adopted 100 rounds in the paper. To make this concrete, we extended induction to 120 rounds and report the relative error reduction of each 20-round interval over the previous checkpoint. Gains on the four distributional metrics diminish as induction proceeds, and the last interval improves them by only about 2% while incurring the same induction cost as earlier ones. Diagnosis accuracy fluctuates across checkpoints, as also shown in Figure 5.

Relative error reduction over the previous checkpoint (%)

RoundsEMR-302 Freq.EMR-302 Cooc.EMR-302 Sim.EMR-302 Cov.EMR-302 Diag.MIMIC-47 Freq.MIMIC-47 Cooc.MIMIC-47 Sim.MIMIC-47 Cov.MIMIC-47 Diag.
0 → 2023.523.734.89.612.314.611.632.27.712.4
20 → 4012.911.518.03.23.810.69.313.03.3-3.6
40 → 609.28.211.01.9-1.36.46.46.61.48.0
60 → 806.25.96.71.0-3.95.95.45.11.0-3.1
80 → 1004.23.38.41.08.24.74.74.10.76.0
100 → 1203.11.13.90.5-4.82.22.02.80.7-6.4
AuthorsOpenReview ↗

Response to Reviewer SCBZ (2/3)

  1. Clinical Assessment by Physicians: Thank you for this suggestion. Three licensed clinicians independently assessed 20 records each from the real corpus, MedSyn (one of the strongest baselines overall in Table 1), and EMRGuide-Zero with Qwen3.5-9B on EMR-302. Without knowing the source, they rated each record for medical errors, internal contradictions, and incompleteness due to missing key information, and judged whether it was real or synthetic. Each rating uses three levels, which we score as 10 (no problem), 5 (minor problem that does not seriously affect clinical use), and 0 (severe problem). Inter-annotator agreement is substantial, with Krippendorff's α of 0.75–0.80 for the three quality ratings and Fleiss' κ of 0.60 for authenticity.
SourceMedical Correctness ↑Consistency ↑Completeness ↑Judged Real ↑
Real8.589.425.9280%
MedSyn8.508.507.8355%
EMRGuide-Zero (ours)9.258.177.9270%

EMRGuide-Zero has the fewest medical errors, comparable consistency and completeness to MedSyn, and is more often judged real (70% vs. 55%), approaching real records (80%). The lower completeness of real records mainly arises because some of their fields are left empty or filled with placeholders. The clinicians also rated 20 sampled guidelines of each type on the same scale. Documentation guidelines score 9.25, and content guidelines score 8.58 with no severe problems. The minor problems in content guidelines mostly concern findings or medications that are not typical of the diagnosis and likely stem from comorbidities. Since content guidelines reproduce how often such content appears in real records rather than requiring it in every record, they remain consistent with the source data.

  1. Extended Evaluation: Thank you for these suggestions. We have extended the evaluation in the following aspects.

Statistical reliability. We computed paired bootstrap 95% CIs for the difference between EMRGuide-Zero and the strongest baseline on each metric, resampling diagnosis groups shared by all methods. Of the 30 comparisons (5 metrics × 2 datasets × 3 backbones), EMRGuide-Zero is significantly better in 19.

Paired Δ (EMRGuide-Zero − strongest baseline) [95% CI] (negative is better for the first four metrics, positive for diagnosis prediction; * marks a significant improvement)

DatasetBackboneEntity Freq.Entity Cooc.Semantic Sim.Semantic Cov.Diagnosis Pred.
EMR-302Qwen3.5-9B-0.0333 [-0.0386, -0.0280]*-0.0562 [-0.0648, -0.0473]*-0.0204 [-0.0246, -0.0163]*-0.0092 [-0.0111, -0.0073]*+0.0116 [+0.0025, +0.0208]*
EMR-302Gemma-4-12B-it-0.0438 [-0.0497, -0.0381]*-0.0597 [-0.0692, -0.0507]*-0.0040 [-0.0083, +0.0001]+0.0034 [+0.0013, +0.0054]+0.0113 [+0.0001, +0.0225]*
EMR-302Llama-3.1-8B-Instruct+0.0076 [+0.0019, +0.0133]+0.0129 [+0.0036, +0.0218]+0.0024 [-0.0010, +0.0057]-0.0002 [-0.0024, +0.0019]+0.0075 [-0.0002, +0.0155]
MIMIC-47Qwen3.5-9B-0.0314 [-0.0513, -0.0142]*-0.0257 [-0.0520, -0.0023]*-0.0265 [-0.0418, -0.0143]*-0.0138 [-0.0207, -0.0081]*+0.0196 [-0.0110, +0.0505]
MIMIC-47Gemma-4-12B-it-0.0695 [-0.0866, -0.0531]*-0.0877 [-0.1098, -0.0663]*-0.0440 [-0.0535, -0.0348]*-0.0229 [-0.0277, -0.0181]*+0.0376 [+0.0101, +0.0657]*
MIMIC-47Llama-3.1-8B-Instruct-0.0177 [-0.0303, -0.0059]*-0.0090 [-0.0270, +0.0082]-0.0042 [-0.0168, +0.0058]-0.0052 [-0.0101, -0.0010]*+0.0082 [-0.0116, +0.0285]
AuthorsOpenReview ↗

Response to Reviewer SCBZ (3/3)

  1. Extended Evaluation (continued)

Downstream evaluation. Using the Qwen3.5-9B-generated corpora, we extend downstream evaluation with a second fine-tuned model, Llama-3.2-1B-Instruct, and three additional tasks: examination and treatment selection following LLM-CARe, and open-ended diagnosis prediction, where the model generates the diagnosis name without options. EMRGuide-Zero achieves the best accuracy in 14 of the 16 task–model–dataset settings, including all settings of open-ended diagnosis prediction, indicating that its utility is not tied to a particular task or downstream model.

DatasetDownstream ModelMethodDiagnosis Selection ↑Examination Selection ↑Treatment Selection ↑Open-Ended Diagnosis ↑
EMR-302Qwen3.5-0.8BDirect0.8210.6050.6520.323
EMR-302Qwen3.5-0.8BMedSyn0.8420.7490.7240.374
EMR-302Qwen3.5-0.8BClinGen0.8170.7560.7430.365
EMR-302Qwen3.5-0.8BLLM-CARe0.8260.7890.7780.352
EMR-302Qwen3.5-0.8BEMRGuide-Zero (ours)0.8540.8120.8110.387
EMR-302Llama-3.2-1B-InstructDirect0.8050.5970.6240.317
EMR-302Llama-3.2-1B-InstructMedSyn0.8300.7390.7010.368
EMR-302Llama-3.2-1B-InstructClinGen0.8100.7300.7250.358
EMR-302Llama-3.2-1B-InstructLLM-CARe0.8140.7690.7510.344
EMR-302Llama-3.2-1B-InstructEMRGuide-Zero (ours)0.8510.7860.7940.387
MIMIC-47Qwen3.5-0.8BDirect0.8070.8880.8890.348
MIMIC-47Qwen3.5-0.8BMedSyn0.8210.9290.9460.382
MIMIC-47Qwen3.5-0.8BClinGen0.8240.9340.9480.399
MIMIC-47Qwen3.5-0.8BLLM-CARe0.7850.9390.9610.393
MIMIC-47Qwen3.5-0.8BEMRGuide-Zero (ours)0.8440.9490.9420.401
MIMIC-47Llama-3.2-1B-InstructDirect0.7860.8850.9070.363
MIMIC-47Llama-3.2-1B-InstructMedSyn0.8190.9110.9460.396
MIMIC-47Llama-3.2-1B-InstructClinGen0.8180.9330.9490.407
MIMIC-47Llama-3.2-1B-InstructLLM-CARe0.8160.9430.9480.390
MIMIC-47Llama-3.2-1B-InstructEMRGuide-Zero (ours)0.8420.9280.9620.420
  1. Implementation of Documentation Guidelines: Thank you for this question. The condition CkC_k and target TkT_k are natural-language criteria proposed together by the Inducer from real–synthetic contrasts, along with the fields they concern. They are executed as record-level binary questions ("Does this record satisfy CkC_k / TkT_k?") answered by the same backbone. Each question concerns a single, narrowly specified property with a yes-or-no answer, which is simpler than open-ended quality assessment. The same check is applied to real and synthetic records, in both validation and refinement during generation. Thus, when the model judges most applicable real records to meet a requirement but fewer synthetic records to do so, the requirement points to a difference between the two corpora under a consistent criterion, which refinement then aims to reduce. In a small manual check of 20 documentation guidelines from the Qwen3.5-9B banks on both datasets, each judged on three real and three synthetic records covering different condition and target outcomes, 81.7% of the judgments were found reasonable.

  2. Information in the Guideline Bank: Thank you for this question. The Inducer views four real records per round when proposing candidates, whereas the Generator receives only the validated guidelines and content statistics, never real record text. Each candidate must recur across broader real records before acceptance, which makes details of individual patients unlikely to enter the bank. Following your suggestion, we further examined the final Qwen3.5-9B banks directly. For content guidelines, since the Inducer proposes guidelines from the records it views, we tested whether these viewed records can be distinguished from other records of the same diagnosis. The AUC is 0.531 on EMR-302 and 0.520 on MIMIC-47, indicating that content guidelines mainly reflect patterns shared across the diagnosis group rather than the particular records used for proposal. For documentation guidelines, which are natural-language text, we ran membership inference against the bank itself. The AUC is 0.501 on EMR-302 and 0.509 on MIMIC-47, close to chance.

Reviewer_AcbVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EMRGuide-Zero, a method to generate synthetic clinical records using LLMs. The method relies on inducing "guidelines" from training data, and then using those guidelines to generate synthetic notes. The process is iterative - guidelines are refined by comparing generations to real data and adjusting guidelines based on that.

The method is evaluated for utility, fidelity and privacy and for the first two, compares favourably against a bunch of baselines. Ablations seem to suggest that the iterative nature of the method indeed improves the quality of notes.

Critical strengths

This is essentially a prompt evolution method, and it seems to be working reasonably well according to the chosen metrics.

Critical weaknesses

A few:

  • Nothing about this method is specific to the medical domain. The approach should be compared to general-domain prompt evolution & data synthesis methods. It is also not clear why a conditional finetuning method is not included, i.e. use labels (diagnosis groups) as conditions to generate the note.

  • Unclear positioning. What is the goal of generating more synthetic data? It's not rare case augmentation or new (out of distribution, e.g. by label combination) case generation - train/test split is in-distribution and rare diagnosis groups are removed. Under this setting, the best-performing baseline would be... ...copy-paste. Admittedly, it wouldn't perform too well under the privacy evaluation, but in the current manuscript, such evaluation is not comprehensive and treated more like an afterthought and the adversaries are too naive. There are established MIA methods for synthetic text, these should be used.

  • Empirical Rigour: The utility evaluation seems too easy - direct (presumably without any knowledge of the training data) generation already achieves 82% accuracy. The four metrics on Entity & semantic similarity both measure content (to a degree), no metric seems to measure form. Semantic similarity measures only the means of features, so diversity is not taken into account. Coverage is better. I don't know why MAUVE wasn't used, which is standard for synthetic text evaluation. The method achieves better "privacy" than "Direct', i.e. presumably without exposure to the training corpus, which makes me question the results. No numbers have uncertainty estimates associated with them.

  • Data contamination: We don't know what either of those three models have been trained on, quite possibly MIMIC-IV was used. Evaluation should include fully-open models (e.g. Olmo) & do a contamination check of the evaluation data on their pre-training corpora (i.e. Dolma for Olmo)

Example of generic prompt evolution baselines:

MIA methods:

Questions for authors

Please address the weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer AcbV (1/3)

  1. Comparison with General Prompt Optimization Methods: We sincerely appreciate your suggestion to compare with general prompt-optimization methods. Compared with general-domain data synthesis, EMR synthesis poses some task-specific requirements, and our design accordingly considers two characteristics of this task.

First, a synthetic EMR corpus needs to cover the variety of real clinical situations. Patients with the same diagnosis can differ in findings, comorbidities, and disease courses, and a corpus concentrated on a few typical presentations would underrepresent other clinical scenarios that downstream models must handle. To preserve this variety, EMRGuide-Zero samples each record's content from frequencies and co-occurrences measured on real records, and applies each documentation guideline only to records that meet its condition.

Second, content observed in real records is not necessarily a requirement that all records should satisfy; it may arise only under a particular patient's circumstances. EMRGuide-Zero therefore validates every candidate guideline before use: content detectors are replayed over the full training set, and documentation guidelines are tested on sampled real records outside the proposal examples, so that details appearing by chance in a few records are not generalized to the whole corpus.

To further examine this, we implemented general-purpose methods for comparison. Aug-PE is built around differential privacy, which our current setting does not adopt, and Rainbow Teaming is designed for generating adversarial prompts. We therefore selected Genetic Prompt, which is also designed for data synthesis and generates diverse data by recombining attributes, and additionally included OPRO as a representative instruction-optimization method, using the distributional similarity between a batch of synthetic records and a validation set of real records as its objective. Both use Qwen3.5-9B with the same conditions, output schema, and corpus size as ours.

DatasetMethodEntity Freq. ↓Entity Cooc. ↓Semantic Sim. ↓Semantic Cov. ↓Diagnosis Pred. ↑
EMR-302OPRO0.2910.4610.1880.2460.817
EMR-302Genetic Prompt0.2650.4160.1680.2350.817
EMR-302EMRGuide-Zero (ours)0.1590.2640.0760.2010.854
MIMIC-47OPRO0.2610.4010.2350.3130.821
MIMIC-47Genetic Prompt0.2690.4180.2840.3190.812
MIMIC-47EMRGuide-Zero (ours)0.1840.3030.1420.2810.844

EMRGuide-Zero outperforms both methods on all metrics across both datasets. OPRO learns a single instruction per diagnosis, which makes it difficult to cover varied clinical situations. Genetic Prompt recombines attributes from the most dissimilar parents, which may combine content arising under different patient conditions into one record. These results highlight the importance of the two considerations in our design for EMR synthesis: covering varied clinical situations and validating guidelines before generalizing them.

  1. Comparison with Conditional Fine-Tuning: Thank you for this suggestion. Following it, we fine-tuned Qwen3.5-9B with LoRA on the training records, conditioned on the diagnosis group and the same demographic inputs, and generated a corpus of the same size.
DatasetMethodEntity Freq. ↓Entity Cooc. ↓Semantic Sim. ↓Semantic Cov. ↓Diagnosis Pred. ↑Membership AUC ↓
EMR-302Conditional Fine-Tuning0.1690.2630.0380.1730.8600.511
EMR-302EMRGuide-Zero (ours)0.1590.2640.0760.2010.8540.507
MIMIC-47Conditional Fine-Tuning0.1620.2470.0510.2190.8630.515
MIMIC-47EMRGuide-Zero (ours)0.1840.3030.1420.2810.8440.504

Because fine-tuning learns directly from real record text, it serves as a strong reference, particularly on semantic metrics. EMRGuide-Zero achieves close diagnosis accuracy (0.854 vs. 0.860 and 0.844 vs. 0.863) and lower entity-frequency error on EMR-302, while its generator never receives real record text. Under the RMIA-calibrated membership attack, EMRGuide-Zero also stays closer to chance. The two approaches further differ in practice: fine-tuning must be repeated for each new generator, whereas our guideline bank is explicit, can be inspected, and transfers across backbones with performance on par with native induction (Figure 4), without updating model parameters.

AuthorsOpenReview ↗

Response to Reviewer AcbV (2/3)

  1. Application Scenario of Synthetic EMRs: Thank you for this question. Following prior work such as MedSyn and LLM-CARe, we target settings where real EMRs are difficult to access or share, and the synthetic corpus serves as a substitute that reproduces the clinical characteristics of the source records for downstream use. The in-distribution evaluation therefore measures how well each method recovers these characteristics. The diagnosis-group filtering in MIMIC-47 only ensures stable per-diagnosis estimates, while EMR-302 retains all 302 groups of the established benchmark, where gains also hold for less frequent diagnoses (Figure 6).

  2. Extended Evaluation: Thank you for the suggestions on evaluation. We have added the following analyses.

Distributional similarity and diversity. We add MAUVE, computed with the same Qwen3-Embedding-0.6B encoder. Diversity is reflected by semantic coverage in Table 1, which measures how closely each real record is matched by its nearest synthetic record; we further add Self-BLEU-4 within each synthetic corpus. EMRGuide-Zero achieves the highest MAUVE and the lowest Self-BLEU in most settings, indicating closer alignment with real records together with greater diversity.

MAUVE ↑

DatasetBackboneDirectMedSynClinGenLLM-CAReEMRGuide-Zero (ours)
EMR-302Qwen3.5-9B0.3150.6090.5170.4190.639
EMR-302Gemma-4-12B-it0.2290.5240.5150.3160.555
EMR-302Llama-3.1-8B-Instruct0.2370.6780.6360.2840.641
MIMIC-47Qwen3.5-9B0.1740.2390.2100.1830.259
MIMIC-47Gemma-4-12B-it0.1690.2040.1970.1750.234
MIMIC-47Llama-3.1-8B-Instruct0.1710.3370.2770.1780.333

Self-BLEU-4 ↓

DatasetBackboneDirectMedSynClinGenLLM-CAReEMRGuide-Zero (ours)
EMR-302Qwen3.5-9B0.9130.6920.8200.7850.681
EMR-302Gemma-4-12B-it0.9280.6910.7610.8070.679
EMR-302Llama-3.1-8B-Instruct0.7190.5210.5510.5980.467
MIMIC-47Qwen3.5-9B0.7500.5350.5040.5480.463
MIMIC-47Gemma-4-12B-it0.8920.6370.6260.7430.571
MIMIC-47Llama-3.1-8B-Instruct0.7350.4100.5010.5310.471

Membership inference. The main paper reports a diagnosis-matched membership AUC (Figure 7), under which EMRGuide-Zero is comparable to Direct and LLM-CARe and below the example-conditioned methods. Following your suggestion, we further apply the RMIA-calibrated similarity attack of Meeus et al. (2025). EMRGuide-Zero remains close to chance across all settings, consistent with the main paper.

DatasetBackboneDirectMedSynClinGenLLM-CAReEMRGuide-Zero (ours)
EMR-302Qwen3.5-9B0.5030.5140.5110.5030.507
EMR-302Gemma-4-12B-it0.5030.5150.5180.5030.510
EMR-302Llama-3.1-8B-Instruct0.5050.5470.5490.5050.511
MIMIC-47Qwen3.5-9B0.5030.5090.5100.5030.504
MIMIC-47Gemma-4-12B-it0.5020.5070.5050.5040.501
MIMIC-47Llama-3.1-8B-Instruct0.5020.5170.5160.5020.501

Statistical reliability. We computed paired bootstrap 95% CIs for the difference between EMRGuide-Zero and the strongest baseline on each metric, resampling diagnosis groups shared by all methods. Of the 30 comparisons (5 metrics × 2 datasets × 3 backbones), EMRGuide-Zero is significantly better in 19.

Paired Δ (EMRGuide-Zero − strongest baseline) [95% CI] (negative is better for the first four metrics, positive for diagnosis prediction; * marks a significant improvement)

DatasetBackboneEntity Freq.Entity Cooc.Semantic Sim.Semantic Cov.Diagnosis Pred.
EMR-302Qwen3.5-9B-0.0333 [-0.0386, -0.0280]*-0.0562 [-0.0648, -0.0473]*-0.0204 [-0.0246, -0.0163]*-0.0092 [-0.0111, -0.0073]*+0.0116 [+0.0025, +0.0208]*
EMR-302Gemma-4-12B-it-0.0438 [-0.0497, -0.0381]*-0.0597 [-0.0692, -0.0507]*-0.0040 [-0.0083, +0.0001]+0.0034 [+0.0013, +0.0054]+0.0113 [+0.0001, +0.0225]*
EMR-302Llama-3.1-8B-Instruct+0.0076 [+0.0019, +0.0133]+0.0129 [+0.0036, +0.0218]+0.0024 [-0.0010, +0.0057]-0.0002 [-0.0024, +0.0019]+0.0075 [-0.0002, +0.0155]
MIMIC-47Qwen3.5-9B-0.0314 [-0.0513, -0.0142]*-0.0257 [-0.0520, -0.0023]*-0.0265 [-0.0418, -0.0143]*-0.0138 [-0.0207, -0.0081]*+0.0196 [-0.0110, +0.0505]
MIMIC-47Gemma-4-12B-it-0.0695 [-0.0866, -0.0531]*-0.0877 [-0.1098, -0.0663]*-0.0440 [-0.0535, -0.0348]*-0.0229 [-0.0277, -0.0181]*+0.0376 [+0.0101, +0.0657]*
MIMIC-47Llama-3.1-8B-Instruct-0.0177 [-0.0303, -0.0059]*-0.0090 [-0.0270, +0.0082]-0.0042 [-0.0168, +0.0058]-0.0052 [-0.0101, -0.0010]*+0.0082 [-0.0116, +0.0285]
AuthorsOpenReview ↗

Response to Reviewer AcbV (3/3)

  1. Robustness to Potential Data Contamination: Thank you for raising this point. Within each setting, all methods share the same backbone, so any contamination would affect them alike rather than favor EMRGuide-Zero. Moreover, Direct, which relies solely on the backbone's prior knowledge, lags clearly behind on every metric. Following your suggestion, we further evaluated with the fully open OLMo-Hybrid-Instruct-DPO-7B on MIMIC-47. Following the 13-gram overlap check of GPT-3, we scanned a random 20% of the shards of its pretraining corpus and found that only 0.01% of 13-grams from MIMIC-47 training records appear in it, indicating negligible contamination. Under this backbone, we compared EMRGuide-Zero with baselines on MIMIC-47, and the results follow the same trend as in the main experiments.
MethodEntity Freq. ↓Entity Cooc. ↓Semantic Sim. ↓Semantic Cov. ↓Diagnosis Pred. ↑
Direct0.2750.4260.2800.3320.765
MedSyn0.2310.3520.1970.3120.814
ClinGen0.2440.3740.1890.3100.805
LLM-CARe0.2340.3550.2140.3140.797
EMRGuide-Zero (ours)0.2310.3670.1750.2980.829
AuthorsOpenReview ↗

A brief summary of the rebuttal

Dear Reviewer AcbV,

Thank you for your time and feedback. Existing EMR synthesis methods struggle to provide fine-grained guidance for diverse clinical scenarios. To address this, our paper introduces EMRGuide-Zero, which automatically induces and validates fine-grained synthesis guidelines, including their frequencies and applicability conditions, to generate higher-quality synthetic EMRs. In response to your comments, our rebuttal provided additional experiments and clarifications, summarized below:

  • Additional baselines: We compared EMRGuide-Zero with general prompt-optimization methods (OPRO and Genetic Prompt) and found that it consistently outperforms them across all metrics. It also achieves performance comparable to conditional fine-tuning without requiring model training, while yielding lower membership leakage and keeping real records unseen by the generator.
  • Extended evaluation: We supplemented our evaluation with metrics for distributional alignment (MAUVE), diversity (Self-BLEU), and privacy (RMIA), along with paired bootstrap significance tests. EMRGuide-Zero performs strongly on distribution and diversity while keeping membership risk close to chance. Furthermore, extending our experiments to the fully open OLMo-Hybrid-Instruct-DPO-7B model confirms negligible data overlap and consistent trends. We have also updated the manuscript to incorporate these new results and analyses.

Thank you again for reviewing our submission. We welcome any further questions or suggestions you may have.

Best regards,

Authors of Submission 58790

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.