acceptodds
Under review as a conference paper at ICLR 2027

PatientLM: Controllable Patient Simulation for Interactive Medical Evaluation and Training

Abstract

Medical assistants are increasingly used in multi-turn consultations, where patients progressively disclose information and adapt their follow-up to assistant responses. However, existing multi-turn evaluation protocols typically rely on fixed dialogue contexts or predefined patient-side trajectories. Such protocols cannot measure how an assistant elicits information or shapes subsequent patient behavior. We introduce **PatientLM**, a patient simulator that combines a trained language model with a controller for specific medical interaction scenarios. PatientLM represents each scenario with a structured patient intent that specifies the patient role and goal, together with the clinical information and requests to disclose over the course of the dialogue. A controller governs this progression, while the trained language model expresses the currently available information naturally in response to the evolving interaction. Simulator evaluations show that PatientLM most completely communicates case-specified information and requests and achieves the strongest overall performance on naturalness and other simulator-quality measures. We then use PatientLM to unify different multi-turn dialogue evaluation paradigms under a common interactive protocol, and experiments show that assistant performance is generally lower than in the original evaluations, suggesting that fixed patient trajectories may overestimate interactive performance. PatientLM also supports effective online SFT for smaller assistant models, improving their factuality and reliability. These results highlight controllable patient simulation as a promising foundation for advancing interactive medical AI.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Language Models, Medical AI, Multi-Turn Dialogue

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_b4LgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PatientLM is a controllable patient simulator that separates dialogue control from utterance generation. Structured personas and disclosure plans guide progressive clinical information release. Experiments show improved case-specific coverage, lower assistant performance under interactive evaluation, and more effective supervision for smaller assistants, particularly through online student-in-the-loop training.

Critical strengths
  1. The method clearly separates dialogue control from patient utterance generation. This design provides a way to manage progressive information disclosure while maintaining a consistent patient role.

  2. The experiments evaluate simulator quality and its usefulness for both assistant evaluation and training.

Critical weaknesses
  1. The DCS metric closely matches the method’s disclosure objective. PatientLM also receives explicit disclosure items that the high-level-intent baseline lacks. The comparison therefore does not isolate the benefits of additional information from those of controller design.

  2. The disclosure plan fixes the information order and prevents voluntary termination before all items are released. These restrictions limit the range of patient behaviors and weaken the connection between controlled case completion and realistic interaction.

  3. The patient model and controller use fixed Qwen backbones, and Qwen3.6-27B also performs several evaluation roles. Testing three assistants does not establish robustness to changes in the patient model, controller, or judge.

  4. The dialogue naturalness evaluation shows substantial position sensitivity. Reference-identification accuracy changes from 76.7% to 30.9% when dialogue order is reversed. The average score being close to chance therefore provides insufficient evidence of realistic patient behavior.

  5. The interactive protocol changes the available context while retaining the original evaluation targets and scoring rules. The paper does not sufficiently validate whether these rules remain appropriate for the generated trajectories. Lower scores alone do not establish that existing benchmarks overestimate real-world performance.

Questions for authors
  1. Could you give all simulators the same clinical information and separately ablate the disclosure plan and controller to identify their individual contributions?

  2. Do the conclusions and assistant rankings remain stable when the patient model, controller, and judge are independently replaced with models from other families?

11 comments

Response to Reviewer b4Lg

We thank the reviewer for the clear and precise review. Q1 in particular gave us a direct design for the ablation that is now the central new experiment of the rebuttal. We address each point below; tables are numbered R1–R9 consistently across all our responses.

W1 / Q1: Equal clinical information, and separate ablations of the plan and the controller

We agree with the reviewer that the effect of the information and that of the controller should be separated, so we ran a finer-grained comparison following exactly the design the reviewer proposes.

Variants (a)–(d) get exactly the same information as PatientLM (✓ in the table): the patient profile (high-level intent) and the ordered list of facts and requests the patient should bring up (disclosure items). They differ only in which patient model writes the turns, and in what decides when the next item is brought up. For reference, the last row is the original UserLM-9B baseline, which gets only the profile (✗).

Table R1. Component ablation on 1,000 HealthBench cases. Nat: how human-like the patient turns are, scored by the EditLens detector (higher is more human-like). "AI": share of patient turns the detector classifies as AI-generated. ID: share of the high-level intent's unigrams (stopwords removed) that reappear in the patient turns; descriptive, as in Table 2. Nat, AI and ID are computed on the patient turns of the interactive HealthBench trajectories (the same trajectories as DCS), so they differ from the offline HealthChat values in Table 2 / Table R3.

VariantDisclosure itemsPatient modelWho decides when to bring up the next itemDCSNatAIID
PatientLM✓trainedcontroller4.8000.7894.2%0.249
(a) Reference schedule✓trainedsame turns as in the original conversation4.7970.7805.0%0.252
(b) One item per turn✓trainedone new item every turn, never stays on a topic4.7720.7874.4%0.246
(c) Prompted generator✓Qwen3.5-9B, prompted onlycontroller4.8720.36572.5%0.371
(d) UserLM-9B + items✓UserLM-9Bthe patient model itself4.6330.66315.4%0.309
UserLM-9B (no items)✗UserLM-9Bthe patient model itself3.3800.68115.4%0.313

Table R1b. DCS by the number of disclosure items per case (number of cases in parentheses).

Variant1 (96)2 (460)3 (269)4 (117)≥5 (58)
PatientLM4.6254.7784.8394.9064.865
(a) Reference schedule4.6464.7744.8494.8874.807
(b) One item per turn4.3544.7844.8394.8914.816
(c) Prompted generator4.9584.8554.9004.9294.614
(d) UserLM-9B + items4.6154.6714.6124.5984.538
UserLM-9B (no items)3.8543.6283.2042.9212.364
  1. Providing the disclosure items brings DCS close to the top of the scale. Every variant that receives them scores between 4.63 and 4.87 out of 5, against 3.38 for UserLM-9B without them. Given the same items, UserLM-9B itself rises from 3.380 to 4.633, which closes about 88% of the 1.42-point gap to PatientLM. Supplying this information is the job of the structured intent, not of the controller.
  2. Releasing the items one at a time matters most when there is a lot to disclose. Variant (d) receives all items at once and decides by itself what to say. Table R1b shows that it falls further behind PatientLM the more items a case has (−0.01, −0.11, −0.23, −0.31 and −0.33 for 1, 2, 3, 4 and ≥5 items), and it copies the item text word for word about twice as often (4-gram copy rate 0.187 vs. 0.086). The variants that release items one at a time with the trained patient model ((a), (b) and PatientLM) stay at 4.77 or above in every group from two items on.
  3. The controller sets this timing from the assistant's replies. Copying the timing of the original conversation (a) and releasing one item per turn (b) give similar coverage (4.797 and 4.772, vs. 4.800 for PatientLM), but neither reacts to what the assistant says. The controller does react: when the assistant refuses or deflects, it stays on the current topic in 71% of cases, but after a complete answer it does so in only 14% (Table R5 below).
  4. Training the patient model makes it sound human; it does not add coverage. With the same controller, a prompted model reaches slightly higher DCS (+0.07), but 72.5% of its turns are classified as AI-generated (Nat 0.365 vs. 0.789), and it repeats more of the profile's wording (ID 0.371 vs. 0.249). Only the trained patient model gets both a high DCS and human-like turns, and it does so whichever of the three schedules (PatientLM, (a), (b)) is used.

Table R5 (also in our response to Reviewer xmwy). How often the patient stays on the current topic (keep) instead of bringing up the next item, by what the assistant did in its previous reply (1,000 interactive HealthBench dialogues). Each assistant reply is labeled with its single main type by a classifier that does not see the patient's next turn.

What the assistant didPatientLM controller(a) Reference schedule(b) One item per turn
Refusal / deflection71%34%0%
Incomplete answer35%28%0%
Clarification request30%13%0%
Targeted question10%12%0%
Complete answer14%27%0%
Offer / options4%11%0%

On DCS matching the method's objective. DCS measures whether the case-specific content reaches the assistant. Benchmark conversion requires this property: if the content is not conveyed, the case's evaluation targets no longer apply. We therefore treat DCS as necessary but not sufficient. Table R1 reports it together with naturalness on the same trajectories, which separates the two: variant (c) has the highest DCS but mostly AI-like turns, so a high DCS alone does not make a good simulator in our evaluation. In addition, DCS is judged by a model from another family in R2, with unchanged rankings.

W2: Fixed order and no voluntary early termination

We agree that both restrictions limit the range of patient behaviors, and that real patients may change the order or end a conversation early. We chose them because they suit automatic conversion and evaluation. The order can be taken directly from each original conversation, whereas a partial order or DAG would require deciding which items truly depend on which, which needs more manual annotation and validation. Releasing all items before termination keeps the case's rubric applicable: if a simulated user stopped before raising a request, the rubric items that respond to that request could not be scored, and scores would no longer be comparable across assistants. Within this structure, the keep action gives some flexibility: the patient can stay on a topic, follow up or react before moving on (Table R5 above). We see relaxing both restrictions as a very promising direction, for example a DAG over items, and voluntary termination with unreleased items excluded from scoring. We will state both restrictions as a limitation and plan to pursue this direction in our future research.

W3 / Q2: Robustness to replacing the patient model, controller and judge

We agree that using models from the same family for generation and evaluation may lead to a family preference. We therefore replaced the judge with a model from another family. For the patient model and the controller, R1 tests how much the results depend on our specific components: replacing the trained patient model with a prompted one (c), or the controller with fixed schedules ((a), (b)), leaves coverage similar.

Judge. We re-judged with Muse-Glimmer-30B, a judge from a different model family, and tested directly whether the Qwen judge favors its own model. The re-judging reuses the original prompts, score parsing and items (2,616 DCS items; 1,000 IA probes per simulator) on the existing trajectories; no dialogue was regenerated.

Table R2. DCS and intent adherence (IA) under the original Qwen3.6-27B judge and under Muse-Glimmer-30B.

SimulatorDCS (Qwen)DCS (Muse)IA (Qwen)IA (Muse)
PatientLM4.8004.5050.9820.973
Qwen3.6-27B3.7373.3710.9930.993
GPT-5.43.6593.2020.9920.989
UserLM-9B3.3802.9920.9480.927
UserLM-8B2.8822.3820.7320.520
PatientSimulator1.9111.5060.6600.646
  • Rankings are almost unchanged. The DCS ranking of all six simulators is the same under both judges. On IA, only the two lowest-scoring simulators swap places, because the Qwen judge is much more lenient toward UserLM-8B (see below).
  • PatientLM's DCS margins grow under Muse: vs. UserLM-9B +1.420 → +1.513 [+1.433, +1.593]; vs. Qwen3.6-27B +1.063 → +1.134 [+1.051, +1.216]; vs. GPT-5.4 +1.141 → +1.303 [+1.217, +1.389] (paired bootstrap). For IA, PatientLM vs. UserLM-9B grows from +0.034 to +0.046 [+0.027, +0.065].
  • No evidence of self-preference. Using Muse as the reference, we checked whether the Qwen judge is more lenient toward the four Qwen-family simulators (its own model Qwen3.6-27B; PatientLM and UserLM-9B, trained from Qwen3.5-9B; PatientSimulator, based on Qwen2.5-72B) than toward simulators from other model families, GPT-5.4 and UserLM-8B (Llama). For each item, we compute the gap, i.e., how much higher the Qwen judge scores than Muse.
    1. The Qwen judge is more lenient overall. On DCS, it scores significantly higher than Muse for every simulator, including the two non-Qwen ones (Table R2b).
    2. It is not more lenient toward Qwen-family simulators on DCS. The gap is smaller for every Qwen-family simulator (+0.28 to +0.38) than for GPT-5.4 (+0.44) and UserLM-8B (+0.49); one-sided Wilcoxon tests, paired by item, for a larger gap on a Qwen-family simulator give p ≥ 0.986 in all eight comparisons, while the tests in the opposite direction give p ≤ 0.014: on DCS, the Qwen judge is in fact significantly less lenient toward Qwen-family simulators than toward the non-Qwen ones (Table R2b).
    3. On intent adherence, there is no consistent preference either (Table R2c). Per simulator, the Qwen judge is significantly more lenient than Muse toward the non-Qwen UserLM-8B, but not toward its own model Qwen3.6-27B, on which the two judges agree exactly (0.993 under both); whether the difference is significant therefore does not follow model family. Against UserLM-8B, toward which the Qwen judge is by far the most lenient (+0.21; 0.732 vs. 0.520), the gap is significantly smaller for all four Qwen-family simulators (p < 0.001). Against GPT-5.4, the gap is not significantly larger for Qwen3.6-27B (p = 0.938) or PatientLM (p = 0.073).
    4. The judge's own model does not lose its lead when the judge is replaced. Qwen3.6-27B's DCS lead over GPT-5.4 grows rather than shrinks under Muse (+0.078 [+0.019, +0.138] → +0.169 [+0.113, +0.225]).
Reviewer_xmwyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Existing medical-assistant evaluations use fixed dialogue contexts or predefined patient-side trajectories, limiting their ability to capture how assistant responses shape subsequent interaction. Adaptive patient simulators must also preserve the clinical information and requests needed to evaluate the same case consistently. The authors introduce PatientLM to address these requirements by combining a trained patient-side language model with structured patient intent and a controller-guided disclosure plan, separating dialogue progression from utterance generation. The experiments demonstrate three main results. First, PatientLM improves disclosure item coverage, achieving a DCS of 4.800 versus 3.380 for the matched high-level-intent-only baseline, alongside favorable automatic naturalness scores. This supports the combined design, although the controller’s individual contribution is not isolated. Second, converting existing benchmarks into an interactive evaluation protocol lowers all three assistants’ HealthBench scores, while KnowMT-Bench results are mixed. This establishes protocol sensitivity rather than proving that fixed trajectories overestimate real-world performance. Third, both offline SFT and student-in-the-loop online SFT improve factual recall and reduce hallucination recall in the tested setting. Online SFT yields lower hallucination recall, while offline SFT yields higher factual recall; the proposed benefit of supervision on student-induced dialogue states remains an unverified explanation.

Critical strengths

1.The matched UserLM-9B comparison provides useful evidence that the combined design improves disclosure coverage: DCS increases from 3.380 to 4.800. This is the paper’s strongest empirical result, although it does not resolve the concerns about component contributions and adaptive behavior.

  1. Preserving evaluation targets and scoring rules when converting existing benchmarks enables informative comparisons across interaction protocols.
Critical weaknesses
  1. The incremental methodological contribution is insufficiently isolated. Prior adaptive virtual patients already separate external state management from utterance generation to control disclosure. The comparison with matched UserLM-9B adds an ordered disclosure plan, state-conditioned patient-model training, and an external controller together. It therefore supports the combined design, but does not establish which components drive the gains or what they contribute beyond existing approaches. This limits the evidence for the claimed benefit of controller-guided information disclosure.

  2. The evaluation establishes coverage more directly than adaptive patient behavior. PatientLM is motivated by the need to preserve case content while adapting patient behavior to assistant responses. Higher DCS demonstrates more complete disclosure of prescribed information and requests, but does not establish that follow-up behavior changes appropriately with assistant responses. Automatic naturalness diagnostics and a qualitative example provide limited evidence for this second requirement. Consequently, the experiments do not yet adequately demonstrate the joint capability central to the paper’s motivation.

Questions for authors
  1. Can you isolate the contributions of the ordered disclosure plan, patient-model training, and dynamic controller?

  2. How would PatientLM respond to controlled variations in assistant behavior, such as targeted questions, misunderstandings, and incomplete answers? Can you evaluate whether the resulting disclosure and follow-up decisions are appropriate, separately from content coverage?

  3. What supports interpreting lower interactive scores as evidence that fixed trajectories “overestimate” interactive competence? Without validation against human interactions, and with wording, available context, dialogue length/history, alignment, and response aggregation changing together, how can protocol effects be distinguished from differences in competence?

  4. How do you separate the benefit of student-induced dialogue states from differences in training data volume and optimization budget? The selected checkpoints occur at steps 58, 174, and 184, and teacher-generated SFT uses right truncation whereas both PatientLM regimes use left truncation. Would the gains persist under matched training-token budgets and consistent truncation? Given that online SFT lowers hallucination recall but also yields lower factual recall than offline SFT, what criterion supports calling it the “best trade-off”?

5 comments

Response to Reviewer xmwy

We thank the reviewer for the detailed review, and in particular for identifying exactly which factors the original comparisons left confounded. We address each point below; tables are numbered R1–R9 consistently across all our responses.

W1 / Q1: Isolating the contributions of the plan, the controller and patient-model training

We agree with the reviewer that the contributions of the information, the controller and patient-model training should be separated, so we ran a finer-grained comparison that isolates each of them.

R1. Component ablation. Variants (a)–(d) get exactly the same information as PatientLM (✓ in the table): the patient profile (high-level intent) and the ordered list of facts and requests the patient should bring up (disclosure items). They differ only in which patient model writes the turns, and in what decides when the next item is brought up. For reference, the last row is the original UserLM-9B baseline, which gets only the profile (✗).

Table R1. Component ablation on 1,000 HealthBench cases. Nat: how human-like the patient turns are, scored by the EditLens detector (higher is more human-like). "AI": share of patient turns the detector classifies as AI-generated. ID: share of the high-level intent's unigrams (stopwords removed) that reappear in the patient turns; descriptive, as in Table 2. Nat, AI and ID are computed on the patient turns of the interactive HealthBench trajectories (the same trajectories as DCS), so they differ from the offline HealthChat values in Table 2 / Table R3.

VariantDisclosure itemsPatient modelWho decides when to bring up the next itemDCSNatAIID
PatientLM✓trainedcontroller4.8000.7894.2%0.249
(a) Reference schedule✓trainedsame turns as in the original conversation4.7970.7805.0%0.252
(b) One item per turn✓trainedone new item every turn, never stays on a topic4.7720.7874.4%0.246
(c) Prompted generator✓Qwen3.5-9B, prompted onlycontroller4.8720.36572.5%0.371
(d) UserLM-9B + items✓UserLM-9Bthe patient model itself4.6330.66315.4%0.309
UserLM-9B (no items)✗UserLM-9Bthe patient model itself3.3800.68115.4%0.313

Table R1b. DCS by the number of disclosure items per case (number of cases in parentheses).

Variant1 (96)2 (460)3 (269)4 (117)≥5 (58)
PatientLM4.6254.7784.8394.9064.865
(a) Reference schedule4.6464.7744.8494.8874.807
(b) One item per turn4.3544.7844.8394.8914.816
(c) Prompted generator4.9584.8554.9004.9294.614
(d) UserLM-9B + items4.6154.6714.6124.5984.538
UserLM-9B (no items)3.8543.6283.2042.9212.364
  1. Providing the disclosure items brings DCS close to the top of the scale. Every variant that receives them scores between 4.63 and 4.87 out of 5, against 3.38 for UserLM-9B without them. Given the same items, UserLM-9B itself rises from 3.380 to 4.633, which closes about 88% of the 1.42-point gap to PatientLM. Supplying this information is the job of the structured intent, not of the controller.
  2. Releasing the items one at a time matters most when there is a lot to disclose. Variant (d) receives all items at once and decides by itself what to say. Table R1b shows that it falls further behind PatientLM the more items a case has (−0.01, −0.11, −0.23, −0.31 and −0.33 for 1, 2, 3, 4 and ≥5 items), and it copies the item text word for word about twice as often (4-gram copy rate 0.187 vs. 0.086). The variants that release items one at a time with the trained patient model ((a), (b) and PatientLM) stay at 4.77 or above in every group from two items on.
  3. The controller sets this timing from the assistant's replies. Copying the timing of the original conversation (a) and releasing one item per turn (b) give similar coverage (4.797 and 4.772, vs. 4.800 for PatientLM), but neither reacts to what the assistant says. The controller does react: when the assistant refuses or deflects, it stays on the current topic in 71% of cases, but after a complete answer it does so in only 14% (R5 below).
  4. Training the patient model makes it sound human; it does not add coverage. With the same controller, a prompted model reaches slightly higher DCS (+0.07), but 72.5% of its turns are classified as AI-generated (Nat 0.365 vs. 0.789), and it repeats more of the profile's wording (ID 0.371 vs. 0.249). Only the trained patient model gets both a high DCS and human-like turns, and it does so whichever of the three schedules (PatientLM, (a), (b)) is used.

Relation to prior adaptive virtual patients. Among prior simulators that use a separate component to decide what the patient reveals, and when, the closest are PatientSim [1] and PatientsWithPersonality (PWP) [2]. We added both to Table 2 and ran them under the same setup as all other simulators (Table R3; full setup in our response to Reviewer AQDJ). PatientLM differs from them in two ways:

  1. What is disclosed. PatientSim and PWP control the release of clinical facts in clinician-led diagnostic interviews. PatientLM's disclosure plan, built from real health-seeking conversations, also carries the user's own requests and concerns. Table R3 shows why this matters: on items that contain only clinical information, PatientSim and PWP do as well as the other baselines (3.24 and 3.03, vs. 3.06 for GPT-5.4), but on items that are the user's own requests they score 0.83 and 1.50 (PatientLM: 4.81), and such requests appear in 90% of the items. Overall DCS is 1.60 for PatientSim and 2.15 for PWP.
  2. Staying natural while being controlled. When a prompted model follows the controller, most of its turns read as AI-written (R1(c)). PatientLM's patient model is trained to follow the controller's instructions while still writing like a real user, so it keeps both coverage and naturalness.

These two properties serve the two requirements of turning an existing non-interactive benchmark into an interactive one. Conveying the whole case, including the user's requests (point 1), preserves the benchmark's original evaluation points, so its targets and scoring rules still apply; as the reviewer notes (S2), this is what makes comparisons across protocols possible. Human-like patient turns (point 2) bring the interaction closer to how real users talk to an assistant.

W2 / Q2: Does follow-up behavior change appropriately with the assistant's response?

A high DCS alone does not show that the patient adapts, so we measured adaptation directly. We analyze the variations in assistant behavior that the reviewer lists (targeted questions and incomplete answers, as well as refusals and clarification requests) as they occur naturally in 1,000 interactive dialogues, and evaluate two things that DCS does not capture: (1) the disclosure decisions and (2) the follow-up reactions. As the reference for appropriate behavior, we use what the users in the original conversations do in the same situation.

How the dialogues are labeled. We took every point in the 1,000 HealthBench dialogues (GPT-5.4 assistant) where the assistant replied and the user then spoke. A classifier first labels what the assistant did, without seeing the user's next turn: asked a targeted question, asked for clarification, gave an incomplete answer, refused or deflected, offered options, or gave a complete answer. A separate call then labels how the user reacted, without seeing the first label. We did this for the original HealthBench conversations and for each simulator (20,970 valid pairs; Qwen3.5-27B as the classifier). Each assistant reply is assigned its single main label.

(1) Disclosure decisions follow the assistant's reply (R5). The table gives, for each kind of assistant reply, how often the patient stays on the current topic (keep) instead of bringing up the next item.

What the assistant didPatientLM controller(a) Reference schedule(b) One item per turn
Refusal / deflection71%34%0%
Incomplete answer35%28%0%
Clarification request30%13%0%
Targeted question10%12%0%
Complete answer14%27%0%
Offer / options4%11%0%

The controller stays on the topic when the assistant has not dealt with it (refusal, incomplete answer, clarification request) and moves on when it has (complete answer, offer); after a refusal or deflection, PatientLM re-asks the unanswered part in 60% of cases. Copying the timing of the original conversation barely depends on what the assistant did, and the fixed rule never stays.

(2) Follow-up reactions resemble the original users and depend on the assistant (R5b). We use two measures: how close each simulator's reactions are to those of the original users (JS), and how strongly its reactions depend on what the assistant did (MI).

Table R5b. Reactions of each simulator compared with those of the users in the original HealthBench conversations. JS asks: when the assistant does the same thing, how different is the simulator's mix of reactions (answering a question, re-asking an unanswered part, following up on the same topic, bringing up something new, accepting an offer, …) from that of the original users? It is the Jensen–Shannon divergence between the two mixes, averaged over the kinds of assistant reply, weighted by how often each occurs in the original conversations; 0 means identical, lower is closer. MI asks: how much does knowing what the assistant did tell us about how the user will react? 0 means the reaction does not depend on the assistant at all; higher means it depends more. It is the normalized mutual information between the kind of assistant reply and the reaction; because this value is slightly above 0 even when the two are unrelated, we subtract the value obtained after randomly shuffling the reactions. Turns in which the user ends the conversation are excluded from both, because they never occur in the original conversations.

SimulatorPatient modelJS ↓MI
Users in the original conversations—00.123
PatientLMtrained0.1010.083
GPT-5.4 prompted as the userGPT-5.40.1440.257
UserLM-9BUserLM-9B0.2890.100
Ablation variants (Table R1)
(a) Reference scheduletrained0.0910.037
(b) One item per turntrained0.0340.044
(d) UserLM-9B + itemsUserLM-9B0.1920.065
  • PatientLM does well on both measures, so its follow-up behavior is both realistic and responsive to the assistant. Its reactions are close to those of the original users (JS 0.101), and they depend on the assistant to a similar degree (MI 0.083 vs. 0.123). The other configurations each fall short on one of the two: GPT-5.4, UserLM-9B and (d) react differently from the original users (JS 0.144–0.289), and (a), (b) and (d) depend less on the assistant (MI 0.037–0.065). GPT-5.4 depends on the assistant even more than the original users (0.257) because it is overly cooperative: it accepts 65% of the assistant's offers.
  • Providing the disclosure items makes the reactions more like the original users'. Giving UserLM-9B the items, as in (d), lowers its JS from 0.289 to 0.192, and PatientLM, which also has them, reaches 0.101. Without the items, UserLM-9B has little to say beyond reacting to the assistant: after a complete answer, it re-asks in 58% of cases, against 3% for the original users and 16% for PatientLM. This is why its MI is slightly higher than PatientLM's (0.100 vs. 0.083): its reactions depend on the assistant, but in a way that is unlike the original users and that conveys less of the case (DCS 3.38). Given all the items at once, however, (d) tends to recite them instead of responding, and its MI falls to 0.065 (it copies item text word for word about twice as often as PatientLM; R1).
  • The controller keeps the patient responsive to the assistant. (a) and (b) have the same information and the same trained patient model as PatientLM; only the controller is replaced. Their reactions remain close to the original users (JS 0.091 and 0.034), but depend on the assistant only about half as much (MI 0.037 and 0.044, vs. 0.083). Releasing the items one at a time, according to what the assistant has just said, is what lets PatientLM benefit from the information without losing its responsiveness to the assistant.

Coverage and adaptation together. Combined with R1, these results show that PatientLM conveys the whole case (DCS 4.80) and, at the same time, adapts its decisions and reactions to the assistant, which is the combination that motivates the paper. The other configurations achieve only one of the two: (a) and (b) cover the case but react to the assistant only about half as much, and GPT-5.4 reacts strongly to the assistant but covers less of the case (DCS 3.66). Its strong reactions also come from being overly cooperative, which makes it less like real users; the UserLM paper [3] reports the same "overly cooperative", sycophantic behavior for assistant LMs prompted to act as users.

Reviewer_AQDJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PatientLM is a patient simulator for medical multi-turn dialogue that pairs a fine-tuned language model with an external controller that governs progressive information disclosure through a structured patient intent containing a high-level patient profile and an ordered disclosure plan. The system is evaluated both as a standalone simulator and as a tool for converting static and dynamic medical benchmarks into interactive evaluation protocols, where it reveals that fixed-trajectory evaluations tend to overestimate assistant performance.

Critical strengths
  • I like the idea of separating dialogue progression (the controller) from utterance generation (the language model). It enables the same case to produce different trajectories, and it also supports interpretability.
  • The dual framing of using the patient simulator both for interactive evaluation and downstream student fine-tuning is not entirely unique, as prior work has either looked at these individually or a close combination thereof (e.g., Ding et al using ProMed for evaluation and reinforcement learning). Nevertheless, to see these applications together in the same paper is nice
  • The evaluations cover comprehensive performance metrics largely informed by UserLM, achieving the best or near-best scores across them
  • The appendices are thorough, providing full prompts and cross-source generalization experiments.
Critical weaknesses
  • PatientsWithPersonality (Schlager et al, 2026) and PatientSim (Kyung et al, 2025) are cited in the related work as directly relevant medical simulators but are not included in Table 2. Without these comparisons, PatientLM is not being compared to the state-of-the-art.
  • The structured patient intents are constructed using LLM prompting. As one of the first steps in the overall approach, errors here propagate downstream to all the other steps. However, this step is never evaluated on its own.
  • Qwen3.6-27B is used as the controller, the DCS judge, the intent-adherence judge, the alignment extractor, and a prompted baseline simulator. This means that the PatientLM is well-calibrated to Qwen3.6-27B's preferences as a judge, making it easier for it to score favorably on DCS. This issue could be mitigated by using an alternative model as a judge (or a jury of judges)
  • For Table 3, I think the method used to generate the labels should be surfaced in the main text. Furthermore, I would like to see the performance metrics split by true positive / true negative rates to support the argument made about why the F1 score for the “keep” action is so low. I don’t think the authors are giving this part enough attention because “keep” turns are one of the aspects that keep the patient simulator more natural.
  • Figure 1 was very unclear to me, especially since it was presented before the notion of the disclosure plan stack was introduced. Without that context, phrases like “action: POP” have little meaning.
Questions for authors
  • Because the disclosure plan is treated as an ordered stack, doesn’t that create a baked-in assumption that simulated patients follow a linear order when providing information? Say for example that we were to provide the disclosure plan illustrated in Figure 2 (symptoms, history, concerns, follow-up). When a patient mentions their symptoms, they may feel compelled to raise the concerns associated with their symptoms if the response they receive compels them to do so, in which case the history would need to be skipped in order to reach concerns. Would a partial ordering or directed acyclic graph be more appropriate (perhaps for future work)?
  • Along similar lines, as the previous question, is it always assumed that simulated patients will always disclose information with depth? Patients are not always so forthright and willing to disclose information immediately, or in some cases at all. My guess is that this could be baked into the role/style components of the high-level intent within the structured patient intent, and this is a critique I have of most patient simulators, but the fact that PatientLM is much better at popping than keeping makes me skeptical that the authors have truly delivered on naturalness as they have claimed
8 comments

Response to Reviewer AQDJ

We thank the reviewer for the positive assessment of the control/generation separation, the dual use for evaluation and training, and the appendices, and for the concrete suggestions. We address each point below; tables are numbered R1–R9 consistently across all our responses.

W1: Comparison with PatientsWithPersonality and PatientSim

We added PatientSim [1] and PatientsWithPersonality (PWP) [2] to Table 2, using the official code and prompts. Both interact with the same GPT-5.4 assistant under the same protocol as all other simulators (at most 15 turns, same retry and termination rules). Since both expect a structured patient record rather than a conversation, we extract their case profiles from each HealthBench conversation using their own extraction prompts (PWP's persona is extracted per case with its HEXACO prompt). All deviations from the original setups are documented in the appendix.

For the backbones, PatientSim uses Llama-3.3-70B-Instruct, the best-performing open-source backbone in the original paper, which was also validated by clinicians. PWP keeps its original meta-model, Ministral-3-14B, which decides which case fields to reveal. Its conversational model, which writes the patient's reply, is Gemini 3.1 Flash-Lite in the original paper; we replace it with Qwen3.5-9B, the same open-source base model as UserLM-9B and PatientLM, so that the comparison does not depend on a stronger or closed backbone.

Table R3. Table 2 with the two new rows (PatientSim and PWP; same metrics and data as the paper). Bold marks the best value in each column; ID is descriptive, and GPT-5.4 perplexity is unavailable through the API.

SimulatorPPL↓FTD↑IDF1_end↑Nat↑RA↑IA↑DCS↑
Qwen3.5-9B18.1470.2230.2180.1740.5640.9530.6402.526
Qwen3.6-27B7.1490.1550.4230.4150.3570.8330.9933.737
GPT-5.4–0.1730.2860.2200.4970.6180.9923.659
UserLM-8B8.1040.2090.3410.3530.7050.8150.7322.882
PatientSimulator36.9830.0680.1240.2110.4170.9020.6601.911
PatientSim (new)17.8910.1210.1410.0710.7110.8740.4491.598
PWP (new)8.3550.1340.2620.1440.4290.5560.7612.148
UserLM-9B4.6590.2470.2710.9240.7550.9950.9483.380
PatientLM4.1640.2470.2240.8900.7710.9960.9824.800

Both simulators are also behind PatientLM on the offline metrics, especially on first-turn diversity (FTD), ending the conversation at the right time (F1_end), and intent adherence (IA).

Why the gap. Both simulators were designed for clinician-led diagnostic interviews, where the patient answers questions about clinical facts. Their case formats hold clinical facts but have no place for the user's own request (e.g., "list five possible causes", "turn this into a SOAP note"), and PatientSim's prompt frames every case as an emergency-department visit. Splitting the 2,616 DCS items by type shows where the difference comes from:

Item type (count)PatientSimPWPGPT-5.4UserLM-9BPatientLM
Clinical information only (249)3.243.033.062.854.95
Information and request (995)1.972.423.583.284.81
Request only (1,372)0.831.503.483.234.81

On clinical information alone, PatientSim and PWP do as well as the other baselines (3.24 and 3.03). Their low overall scores come from the request items, which make up 90% of the items in health-seeking conversations. PatientLM's disclosure plan is built from such conversations and carries both the clinical information and the user's requests.

W2: Evaluating structured-intent construction on its own

R8. Human audit. We randomly sampled 100 cases (50 HealthBench, 50 HealthChat; fixed seed) from the evaluation sets used in the paper. One of the authors annotated all cases, checking every LLM-generated part of the structured intent against the original conversation and following a written guideline:

  • Disclosure items. For each user turn, the annotator recorded content that the user said but the disclosure item left out (omission), and content in the item that the conversation does not support (hallucination). Each was graded 0 (none), 1 (minor: a detail such as a number or frequency is lost, or an inference is added) or 2 (severe: a core fact or request is lost, or a clinical fact is invented).
  • Turn types. For HealthBench, our LLM labels each user turn as context (A), request (B), both, communication management (C) or off-topic (D); we compare these labels with the human labels. For the controller, the five types reduce to three: a context or request turn (A, B or both) creates a new disclosure item (pop), a communication-management turn (C) stays on the current item (keep), and an off-topic turn (D) is left out of the plan. We therefore also report agreement on these three classes. HealthChat cases use the labels released with the dataset.
  • High-level profile. For each case, the annotator checked whether the profile gives away concrete clinical facts that should only come out later as disclosure items (0 none / 1 scenario-level, e.g., naming a later diagnosis / 2 detailed, e.g., values or doses), and whether the profile fits the original conversation (2 fits / 1 minor issues, but role and goal correct / 0 wrong role or goal, or invented content).

Table R8. Human audit of structured-intent construction (100 cases).

CheckResult
Turn types: agreement with human labels (HealthBench, 141 turns)Cohen's κ 0.82 (5 types); 0.86 (3 classes: pop / keep / left out)
User turns with any omission / severe omission12.3% / 6.4% (of 235)
Disclosure items with any hallucination / severe hallucination2.8% / 0% (of 217)
Profiles that give away a later fact: scenario-level / detailed3% / 0% (of 100)
Profile fit: fits / minor issues / wrong86% / 12% / 2% (of 100)

The omission rate counts every HealthBench user turn, including turns where a wrong C/D label meant that no item was created, and every HealthChat turn that has an item. HealthChat turns without an item are excluded, because their labels come from the dataset rather than from our pipeline.

  • The LLM almost never invents content. No disclosure item invents a clinical fact, and only 2.8% contain a minor over-interpretation, e.g., adding "seeking reassurance" to a purely emotional statement. No profile gives away details of later items; 3 of 100 mention a later diagnosis at the scenario level.
  • The main error is omission. 6.4% of user turns lose a core fact or request, either because the turn was labeled as communication management or off-topic so no item was created, or because a key detail was compressed away (e.g., the user's own incorrect claim that the assistant is expected to correct).
  • Turn-type labels agree well with humans (κ=0.82 over the five types; 0.86 over the three classes that determine the disclosure plan).

Errors in this step therefore lead mainly to missing content rather than invented content.

W3: Qwen3.6-27B in several roles

We agree that using Qwen3.6-27B both in the simulation and as the judge may introduce a preference toward PatientLM. To check this, we ran additional experiments: we re-judged with Muse-Glimmer-30B, a judge from a different model family, and tested directly whether the Qwen judge favors its own model. The re-judging reuses the original prompts, score parsing and items (2,616 DCS items; 1,000 IA probes per simulator) on the existing trajectories; no dialogue was regenerated.

Table R2. DCS and intent adherence (IA) under the original Qwen3.6-27B judge and under Muse-Glimmer-30B.

SimulatorDCS (Qwen)DCS (Muse)IA (Qwen)IA (Muse)
PatientLM4.8004.5050.9820.973
Qwen3.6-27B3.7373.3710.9930.993
GPT-5.43.6593.2020.9920.989
UserLM-9B3.3802.9920.9480.927
UserLM-8B2.8822.3820.7320.520
PatientSimulator1.9111.5060.6600.646
  • Rankings are almost unchanged. The DCS ranking of all six simulators is the same under both judges. On IA, only the two lowest-scoring simulators swap places, because the Qwen judge is much more lenient toward UserLM-8B (see below).
  • PatientLM's DCS margins grow under Muse: vs. UserLM-9B +1.420 → +1.513 [+1.433, +1.593]; vs. Qwen3.6-27B +1.063 → +1.134 [+1.051, +1.216]; vs. GPT-5.4 +1.141 → +1.303 [+1.217, +1.389] (paired bootstrap). For IA, PatientLM vs. UserLM-9B grows from +0.034 to +0.046 [+0.027, +0.065].
  • No evidence of self-preference. Using Muse as the reference, we checked whether the Qwen judge is more lenient toward the four Qwen-family simulators (its own model Qwen3.6-27B; PatientLM and UserLM-9B, trained from Qwen3.5-9B; PatientSimulator, based on Qwen2.5-72B) than toward simulators from other model families, GPT-5.4 and UserLM-8B (Llama). For each item, we compute the gap, i.e., how much higher the Qwen judge scores than Muse.
    1. The Qwen judge is more lenient overall. On DCS, it scores significantly higher than Muse for every simulator, including the two non-Qwen ones (Table R2b).
    2. It is not more lenient toward Qwen-family simulators on DCS. The gap is smaller for every Qwen-family simulator (+0.28 to +0.38) than for GPT-5.4 (+0.44) and UserLM-8B (+0.49); one-sided Wilcoxon tests, paired by item, for a larger gap on a Qwen-family simulator give p ≥ 0.986 in all eight comparisons, while the tests in the opposite direction give p ≤ 0.014: on DCS, the Qwen judge is in fact significantly less lenient toward Qwen-family simulators than toward the non-Qwen ones (Table R2b).
    3. On intent adherence, there is no consistent preference either (Table R2c). Per simulator, the Qwen judge is significantly more lenient than Muse toward the non-Qwen UserLM-8B, but not toward its own model Qwen3.6-27B, on which the two judges agree exactly (0.993 under both); whether the difference is significant therefore does not follow model family. Against UserLM-8B, toward which the Qwen judge is by far the most lenient (+0.21; 0.732 vs. 0.520), the gap is significantly smaller for all four Qwen-family simulators (p < 0.001). Against GPT-5.4, the gap is not significantly larger for Qwen3.6-27B (p = 0.938) or PatientLM (p = 0.073).
    4. The judge's own model does not lose its lead when the judge is replaced. Qwen3.6-27B's DCS lead over GPT-5.4 grows rather than shrinks under Muse (+0.078 [+0.019, +0.138] → +0.169 [+0.113, +0.225]).

Comments on the paper

4 comments

General Response to All Reviewers

We thank all reviewers for their careful and constructive feedback. Reviewers found that separating dialogue control from utterance generation is clear, supports interpretability and lets one case yield multiple trajectories (AQDJ, b4Lg); that the matched UserLM-9B comparison is the paper's strongest empirical result (xmwy); that keeping evaluation targets and scoring rules fixed enables comparison across protocols (xmwy); and that the evaluation covers simulator quality as well as use for evaluation and training (AQDJ, b4Lg).

We ran new experiments for every concern that could be tested in the rebuttal period. Full details, tables and statistics are in the individual responses. Tables are numbered R1–R9 consistently across all responses.

New experimentDiscussed in responses to
R1Component ablation with equal clinical information (4 new variants × 1,000 cases), plus naturalness on the same trajectoriesxmwy W1/Q1, b4Lg W1/Q1
R2DCS and intent adherence re-judged by a non-Qwen judge (Muse-Glimmer-30B)AQDJ W3, b4Lg W3/Q2
R3PatientSim and PatientsWithPersonality added to Table 2AQDJ W1, xmwy W1
R4Faithfulness-controlled comparison of static vs. interactive HealthBench scoresxmwy Q3, b4Lg W5
R5Controller decisions and user reactions conditioned on the assistant's behaviorxmwy W2/Q2, AQDJ Q2, b4Lg W1/Q1
R6Teacher-forcing SFT with matched truncation, optimizer steps and token budgetxmwy Q4
R7Per-action TP/TN breakdown of Table 3, with the label sourceAQDJ W4
R8Human audit of structured-intent constructionAQDJ W2
R9Position-debiased real-vs-simulated identificationb4Lg W4

Main findings.

  1. Where the gains come from (R1). We gave all variants the same information and varied the other components.
    • Information explains most of the DCS gap: with the disclosure items, every variant scores between 4.63 and 4.87 out of 5, close to the top of the scale. UserLM-9B itself rises from 3.380 to 4.633, closing about 88% of the gap to PatientLM.
    • Releasing the items one at a time explains most of the rest. When the model receives all items at once and decides by itself what to say, it falls further behind as cases get longer (−0.31 DCS at four or more items). When the items are released one at a time, DCS stays around 4.8.
    • The learned controller sets this timing from the assistant's replies: when the assistant refuses or deflects, it stays on the current topic in 71% of cases, but after a complete answer it does so in only 14%, whereas copying the timing of the original conversation or a fixed rule barely responds to the assistant (R5).
    • Training the patient model makes the turns human-like: with the same controller, a prompted model reaches similar coverage, but 72.5% of its turns are classified as AI-generated, against 4.2% for PatientLM.
  2. Robustness to the judge (R2). Since using models from the same family for generation and evaluation may lead to a family preference, we re-judged DCS and intent adherence with Muse-Glimmer-30B, a judge from a different model family. The DCS ranking of all six simulators is unchanged, PatientLM's DCS margins over all baselines grow, and on intent adherence only the two lowest-scoring simulators swap places. We also tested directly whether the Qwen judge favors Qwen-family simulators. Using Muse as the reference, we measured for every item how much more leniently the Qwen judge scores than Muse. The Qwen judge is more lenient than Muse overall, but not more so toward the four Qwen-family simulators (its own model Qwen3.6-27B, PatientLM, UserLM-9B and PatientSimulator). On DCS, their gaps (+0.28 to +0.38) are all smaller than those of GPT-5.4 (+0.44) and UserLM-8B (+0.49): one-sided paired Wilcoxon tests for a larger gap give p ≥ 0.986 in all eight comparisons. On intent adherence, the two judges agree exactly on Qwen3.6-27B, and the Qwen judge is by far the most lenient toward the Llama-based UserLM-8B.
  3. State-of-the-art baselines (R3). PatientSim and PWP were designed for clinician-led diagnostic interviews, where the patient answers questions about clinical facts. On items that contain only clinical information, PatientSim and PWP do as well as the other baselines (3.24 and 3.03, vs. 3.06 for GPT-5.4 and 2.85 for UserLM-9B). In health-seeking conversations, however, 90% of the items include the user's own request (e.g., "turn this into a SOAP note"), which their case format has no place for. On request-only items, they score 0.83 and 1.50, against 4.81 for PatientLM. Overall DCS is 1.60 for PatientSim and 2.15 for PWP, against 4.80 for PatientLM, and both also fall behind PatientLM on the offline metrics. PatientSim uses Llama-3.3-70B, the best open-source backbone in its original paper; for PWP, we keep its original meta-model and replace its conversational model with Qwen3.5-9B, the same base model as PatientLM.
  4. Static vs. interactive evaluation (R4). To isolate the effect of the interaction itself, we ran a more controlled comparison. We keep only cases where (a) for all three assistants, the simulated user preserves the original request and every rubric-relevant fact, and (b) the request is asked in a single turn, so that a single reply is scored. Check (a) is done by Muse-Glimmer-30B: because the simulator and its controller are built on Qwen models, we use a judge from a different model family, so that a judge from the simulator's own family would not be too lenient when deciding whether the simulated turns are faithful. On the resulting 207 cases, each interactive reply is graded exactly as in the official HealthBench evaluation, with the same original conversation as context, the same rubric and the same grader, so the rubric applies to it just as it does to the original reply. The only difference is that the assistant reached its reply by conducting the conversation itself. Interactive scores remain significantly lower (pooled Δ=−0.053, p=0.0001). We will restrict the claim to "static evaluation tends to overestimate how well assistants perform in interactive consultations".
  5. Adaptivity and SFT (R5, R6). The controller's decisions depend on the assistant's behavior: when the assistant refuses or deflects, it stays on the current topic in 71% of cases, against 14% after a complete answer. The SFT advantage over teacher forcing persists under matched truncation, optimizer steps and token budgets.

Revisions in the updated PDF (all marked in blue).

  • R1 Component ablation: new Section 4.3 with Table 4.
  • R2 Judge robustness: Appendix C.3 (Tables 12–13), summarized at the end of Section 4.2.
  • R3 PatientSim and PWP: two new rows in Table 2 and in the baseline list of Section 4.1; setup and item-type breakdown in Appendix C.2 (Table 11).
  • R4 Faithfulness-controlled comparison: Section 5.1 and Appendix C.7 (Table 18). The "overestimate" claim is restated in this scope in the Abstract, Section 1 and Section 5.1.
  • R5 Adaptivity analysis: Appendix C.4 (Tables 14–15), summarized in Section 4.3.
  • R6 Teacher-generated SFT retrained under matched settings: Table 7 (Section 5.2) now reports its best run, and all epochs are in Appendix C.8 (Table 19). The "best trade-off" wording is removed from Section 1 and Section 5.2.
  • R7 Precision, TPR and TNR added to Table 3; the label source is cited in Section 4.4 and detailed, with the per-action analysis, in Appendix C.5 (Table 16).
  • R8 Intent audit: Appendix C.6 (Table 17), summarized in Section 2.1.
  • R9 Position-debiased analysis: Appendix C.1 (Table 10); "near-chance" is replaced by "close to chance and far below the baselines".
  • Figure 1 is redrawn to be self-contained.
  • The fixed disclosure order and the absence of voluntary early termination are stated as a limitation in Section 6.

We are grateful for the reviewers' suggestions, which substantially improved the analysis, and we are happy to answer any further questions.

Area_Chair_NyhjOpenReview ↗

Author-Reviewer Discussion

Dear Reviewers:

Thank you for your thoughtful reviews. The authors have responded to your reviews. Please check their response and reach out to them if you have any further questions. Please act as soon as possible, as the discussion period with the authors closes on Nov 18.

Best regards,

  • AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.