- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PatientLM is a controllable patient simulator that separates dialogue control from utterance generation. Structured personas and disclosure plans guide progressive clinical information release. Experiments show improved case-specific coverage, lower assistant performance under interactive evaluation, and more effective supervision for smaller assistants, particularly through online student-in-the-loop training.
Critical strengths
-
The method clearly separates dialogue control from patient utterance generation. This design provides a way to manage progressive information disclosure while maintaining a consistent patient role.
-
The experiments evaluate simulator quality and its usefulness for both assistant evaluation and training.
Critical weaknesses
-
The DCS metric closely matches the method’s disclosure objective. PatientLM also receives explicit disclosure items that the high-level-intent baseline lacks. The comparison therefore does not isolate the benefits of additional information from those of controller design.
-
The disclosure plan fixes the information order and prevents voluntary termination before all items are released. These restrictions limit the range of patient behaviors and weaken the connection between controlled case completion and realistic interaction.
-
The patient model and controller use fixed Qwen backbones, and Qwen3.6-27B also performs several evaluation roles. Testing three assistants does not establish robustness to changes in the patient model, controller, or judge.
-
The dialogue naturalness evaluation shows substantial position sensitivity. Reference-identification accuracy changes from 76.7% to 30.9% when dialogue order is reversed. The average score being close to chance therefore provides insufficient evidence of realistic patient behavior.
-
The interactive protocol changes the available context while retaining the original evaluation targets and scoring rules. The paper does not sufficiently validate whether these rules remain appropriate for the generated trajectories. Lower scores alone do not establish that existing benchmarks overestimate real-world performance.
Questions for authors
-
Could you give all simulators the same clinical information and separately ablate the disclosure plan and controller to identify their individual contributions?
-
Do the conclusions and assistant rankings remain stable when the patient model, controller, and judge are independently replaced with models from other families?
11 comments
Table R5 (also in our response to Reviewer xmwy). How often the patient stays on the current topic (keep) instead of bringing up the next item, by what the assistant did in its previous reply (1,000 interactive HealthBench dialogues). Each assistant reply is labeled with its single main type by a classifier that does not see the patient's next turn.
| What the assistant did | PatientLM controller | (a) Reference schedule | (b) One item per turn |
|---|---|---|---|
| Refusal / deflection | 71% | 34% | 0% |
| Incomplete answer | 35% | 28% | 0% |
| Clarification request | 30% | 13% | 0% |
| Targeted question | 10% | 12% | 0% |
| Complete answer | 14% | 27% | 0% |
| Offer / options | 4% | 11% | 0% |
On DCS matching the method's objective. DCS measures whether the case-specific content reaches the assistant. Benchmark conversion requires this property: if the content is not conveyed, the case's evaluation targets no longer apply. We therefore treat DCS as necessary but not sufficient. Table R1 reports it together with naturalness on the same trajectories, which separates the two: variant (c) has the highest DCS but mostly AI-like turns, so a high DCS alone does not make a good simulator in our evaluation. In addition, DCS is judged by a model from another family in R2, with unchanged rankings.
W2: Fixed order and no voluntary early termination
We agree that both restrictions limit the range of patient behaviors, and that real patients may change the order or end a conversation early. We chose them because they suit automatic conversion and evaluation. The order can be taken directly from each original conversation, whereas a partial order or DAG would require deciding which items truly depend on which, which needs more manual annotation and validation. Releasing all items before termination keeps the case's rubric applicable: if a simulated user stopped before raising a request, the rubric items that respond to that request could not be scored, and scores would no longer be comparable across assistants. Within this structure, the keep action gives some flexibility: the patient can stay on a topic, follow up or react before moving on (Table R5 above). We see relaxing both restrictions as a very promising direction, for example a DAG over items, and voluntary termination with unreleased items excluded from scoring. We will state both restrictions as a limitation and plan to pursue this direction in our future research.
W3 / Q2: Robustness to replacing the patient model, controller and judge
We agree that using models from the same family for generation and evaluation may lead to a family preference. We therefore replaced the judge with a model from another family. For the patient model and the controller, R1 tests how much the results depend on our specific components: replacing the trained patient model with a prompted one (c), or the controller with fixed schedules ((a), (b)), leaves coverage similar.
Judge. We re-judged with Muse-Glimmer-30B, a judge from a different model family, and tested directly whether the Qwen judge favors its own model. The re-judging reuses the original prompts, score parsing and items (2,616 DCS items; 1,000 IA probes per simulator) on the existing trajectories; no dialogue was regenerated.
Table R2. DCS and intent adherence (IA) under the original Qwen3.6-27B judge and under Muse-Glimmer-30B.
| Simulator | DCS (Qwen) | DCS (Muse) | IA (Qwen) | IA (Muse) |
|---|---|---|---|---|
| PatientLM | 4.800 | 4.505 | 0.982 | 0.973 |
| Qwen3.6-27B | 3.737 | 3.371 | 0.993 | 0.993 |
| GPT-5.4 | 3.659 | 3.202 | 0.992 | 0.989 |
| UserLM-9B | 3.380 | 2.992 | 0.948 | 0.927 |
| UserLM-8B | 2.882 | 2.382 | 0.732 | 0.520 |
| PatientSimulator | 1.911 | 1.506 | 0.660 | 0.646 |
- Rankings are almost unchanged. The DCS ranking of all six simulators is the same under both judges. On IA, only the two lowest-scoring simulators swap places, because the Qwen judge is much more lenient toward UserLM-8B (see below).
- PatientLM's DCS margins grow under Muse: vs. UserLM-9B +1.420 → +1.513 [+1.433, +1.593]; vs. Qwen3.6-27B +1.063 → +1.134 [+1.051, +1.216]; vs. GPT-5.4 +1.141 → +1.303 [+1.217, +1.389] (paired bootstrap). For IA, PatientLM vs. UserLM-9B grows from +0.034 to +0.046 [+0.027, +0.065].
- No evidence of self-preference. Using Muse as the reference, we checked whether the Qwen judge is more lenient toward the four Qwen-family simulators (its own model Qwen3.6-27B; PatientLM and UserLM-9B, trained from Qwen3.5-9B; PatientSimulator, based on Qwen2.5-72B) than toward simulators from other model families, GPT-5.4 and UserLM-8B (Llama). For each item, we compute the gap, i.e., how much higher the Qwen judge scores than Muse.
- The Qwen judge is more lenient overall. On DCS, it scores significantly higher than Muse for every simulator, including the two non-Qwen ones (Table R2b).
- It is not more lenient toward Qwen-family simulators on DCS. The gap is smaller for every Qwen-family simulator (+0.28 to +0.38) than for GPT-5.4 (+0.44) and UserLM-8B (+0.49); one-sided Wilcoxon tests, paired by item, for a larger gap on a Qwen-family simulator give p ≥ 0.986 in all eight comparisons, while the tests in the opposite direction give p ≤ 0.014: on DCS, the Qwen judge is in fact significantly less lenient toward Qwen-family simulators than toward the non-Qwen ones (Table R2b).
- On intent adherence, there is no consistent preference either (Table R2c). Per simulator, the Qwen judge is significantly more lenient than Muse toward the non-Qwen UserLM-8B, but not toward its own model Qwen3.6-27B, on which the two judges agree exactly (0.993 under both); whether the difference is significant therefore does not follow model family. Against UserLM-8B, toward which the Qwen judge is by far the most lenient (+0.21; 0.732 vs. 0.520), the gap is significantly smaller for all four Qwen-family simulators (p < 0.001). Against GPT-5.4, the gap is not significantly larger for Qwen3.6-27B (p = 0.938) or PatientLM (p = 0.073).
- The judge's own model does not lose its lead when the judge is replaced. Qwen3.6-27B's DCS lead over GPT-5.4 grows rather than shrinks under Muse (+0.078 [+0.019, +0.138] → +0.169 [+0.113, +0.225]).
Response to Reviewer b4Lg
We thank the reviewer for the clear and precise review. Q1 in particular gave us a direct design for the ablation that is now the central new experiment of the rebuttal. We address each point below; tables are numbered R1–R9 consistently across all our responses.
W1 / Q1: Equal clinical information, and separate ablations of the plan and the controller
We agree with the reviewer that the effect of the information and that of the controller should be separated, so we ran a finer-grained comparison following exactly the design the reviewer proposes.
Variants (a)–(d) get exactly the same information as PatientLM (✓ in the table): the patient profile (high-level intent) and the ordered list of facts and requests the patient should bring up (disclosure items). They differ only in which patient model writes the turns, and in what decides when the next item is brought up. For reference, the last row is the original UserLM-9B baseline, which gets only the profile (✗).
Table R1. Component ablation on 1,000 HealthBench cases. Nat: how human-like the patient turns are, scored by the EditLens detector (higher is more human-like). "AI": share of patient turns the detector classifies as AI-generated. ID: share of the high-level intent's unigrams (stopwords removed) that reappear in the patient turns; descriptive, as in Table 2. Nat, AI and ID are computed on the patient turns of the interactive HealthBench trajectories (the same trajectories as DCS), so they differ from the offline HealthChat values in Table 2 / Table R3.
Table R1b. DCS by the number of disclosure items per case (number of cases in parentheses).