- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on the limitations of existing ideal patient simulations, arguing that these patients are overly cooperative and do not adequately simulate real-world patients who may withhold information due to shame, privacy concerns, or similar reasons. The authors therefore design a simulated patient that selectively discloses information based on trust, information sensitivity, and an overall willingness to cooperate. This patient continuously updates its internal trust state during conversations with the doctor agent, then uses predefined algorithms and random sampling to select a disclosure strategy, aiming to simulate some non-ideal patients in real-world settings. The authors also construct S²InQBench from public medical datasets for use with the patient simulation framework. Finally, they evaluate 13 large language models and find that, in cases where diagnosis depends on sensitive information, all tested models achieve lower diagnostic accuracy with patients who conceal information than under the ideal patient condition.
Critical strengths
1. The methodology and implementation show a degree of rigor, combining traditional NLP and statistical methods at several key steps rather than relying entirely on direct LLM judgments. For example, during CSU normalization, BioLinkBERT-Large converts concepts and their associated descriptions into embeddings, which are then clustered using Agglomerative Hierarchical Clustering (AHC). The Bradley–Terry model is also used to further calibrate CSU sensitivity scores based on the initial ratings provided by LLMs.
2. The paper contributes to patient simulation research by providing a method for simulating patients who do not fully disclose information.
Critical weaknesses
1. The paper’s motivation and contribution may not be as strong as the authors suggest.
1.1. The clinical representativeness of the patient disclosure mechanism requires further validation.
Existing empirical research shows that patients are less likely to report that they would refuse to disclose their sexual orientation than healthcare professionals expect [1]. Another study found that most survey participants answered questions about sexual orientation and gender identity. This suggests that patients do not necessarily refuse to cooperate even when personal information is involved [2].
In practice, physicians and patients may more often work together toward the same goal. Therefore, I believe that the adversarial interactions considered in this paper may not represent most clinical encounters. The authors describe patients in prior work as “ideal patients” because they provide information in response to queries. However, could the uncooperative patients proposed in this paper represent another type of “extreme” patient? For this reason, I find the paper’s motivation and contribution less compelling than the authors suggest.
1.2. The paper overgeneralizes prior work on patient simulation.
There is already related work on simulating patients who conceal information. Existing studies have considered hidden information, selective disclosure, and untruthful responses. Examples include MedConceal [3], MedDialBench [4], and Patients With Personality [5].
[1] https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2621833
[2] https://pubmed.ncbi.nlm.nih.gov/30665975/
[3] https://arxiv.org/abs/2604.08788
[4] https://arxiv.org/abs/2604.06846
[5] https://arxiv.org/abs/2606.17441
2. The experimental design has shortcomings.
2.1. The hyperparameter settings for the patient simulator are insufficiently justified.
In Sec. 3.2, the negative trust update coefficient is set to three times the positive one. This setting, along with the trespass penalty and memory retention factor, directly affects the results. However, the paper does not provide sufficient parameter sensitivity analysis or component ablations. The possibility that the model rankings are mainly driven by these settings needs to be ruled out.
2.2. Disclosure willingness is confounded with proactive information sharing.
According to the “Ideal Patient Agent — System Prompt” in App. H.2, the ideal patient’s task includes:
Answer the doctor’s questions truthfully, completely and directly, as a fully cooperative patient who never conceals, softens, evades, distorts or "forgets" any part of the history, including embarrassing or stigmatized facts.
It also requires:
Proactively mention every fact in your history that is relevant to the doctor’s current question, even if only partly asked.
Meanwhile, the constraints in the “Stigma-Aware Patient Agent — System Prompt” state:
Answer only what the doctor asks; do not volunteer extra information that was not asked about.
Therefore, the performance gap may reflect differences in both “willingness to disclose information” and “how proactively patients provide information,” making it difficult to attribute the gap entirely to the stigma mechanism.
Questions for authors
-
For readability, would you consider adding a pipeline diagram to the methods section to illustrate the approach more intuitively, rather than explaining it mainly through a large number of equations?
-
Could you increase the font sizes in figures such as Fig. 2 and Fig. 3?
-
Could you provide appropriate statistical significance tests for the results?
-
Could you add case studies of specific dialogues and a more detailed error analysis to further examine why model performance degrades when interacting with simulated patients who selectively disclose information?
14 comments
Response to Weakness 1.1: Motivation, Clinical Representativeness, and Nature of Patient Interaction (2/2)
3. Our Patient Agent is NOT "Another Extreme"
The reviewer doubts whether our patient represents an "uncooperative extreme," opposite to prior "ideal patients." Empirical data from our benchmark demonstrates that captures a realistic behavioral continuum, not an uncooperative caricature:
- Cooperation is Dynamically Earned, Not Statically Blocked: Disclosure is responsive to clinician bedside manner rather than permanently withheld. Resolved by probe index across sensitive inquiries (Table 20), the patient agent's full disclosure jumps by +12.1 pp from the initial to the second sensitive probe (21.2% 33.3%, plateauing around ~31%), as the doctor establishes a therapeutic alliance. At this stage, truthful disclosure substantially outstrips evasive deflection (ER, 24.6%) and fabrication (DF, 8.7%), demonstrating that the agent actively cooperates once psychological safety is signaled.
- Partial Omission (PO) is the Modal Posture, Not Flat Refusal: Across the high-stigma probing on Core cases (), the dominant response posture is Partial Omission (PO, 43.6%), where patients truthfully describe surface somatic complaints while omitting the stigmatizing etiology (Table 16), followed by Evasion (ER, 23.8%) and fabrication (DF, 10.7%). This faithfully mirrors a common real-world patient behavior: answering the physical symptom while withholding the embarrassing backstory.
- Rigorous Validation Against Human Patients: In our blind human validation study across 50 stratified Core cases (Appendix I.1, Table 20), human participants disclosed fully on only 27.1% of initial sensitive probes, with 64.7% resorting to partial omission or redirection. Our agent matched human disclosure dynamics with an aggregate Total Variation Distance of across the first five sensitive probes (well within the predefined 0.10 margin). For transparency, 27.1% is the first-probe rate; pooling all five probe positions by consultation count, full disclosure is 31.2% for humans and 27.5% for the agent, and this 3.7-pp difference also accounts for the aggregate TVD of 0.037 when considering the full four-posture distributions. The agent is therefore marginally more withholding than a human patient, while tracking the human posture dynamics (FD rise, PO takeover, ER collapse) closely. The agent's hesitation also mirrors human behavior under identical clinical scenarios.
4. Manuscript Revision
We sincerely appreciate the reviewer’s perspective regarding terminology. To prevent any unintended perception that we are replacing one extreme with another, we will replace the shorthand term "ideal patient" with "fully cooperative patient" throughout the manuscript and explicitly characterize our agent as a "trust-gated, conditionally disclosive patient" to distinguish it from an adversarial opponent.
These refinements will further sharpen the positioning of as an ecologically grounded evaluation of clinical bedside communication.
Response to Weakness 1.2: Novelty and Relationship to Prior Work (1/3)
We sincerely thank the reviewer for raising this critical discussion regarding prior works on patient simulation (MedConceal [6], MedDialBench [7], and PWP [8]). We appreciate the opportunity to clarify the fundamental distinctions in scientific objectives, problem formulation, and underlying computational mechanisms between and these works.
1. Novelties are Distinguished by Research Question and Evaluation Target
We first clarify that is not yet another realistic patient simulator. Rather, patient simulation is strictly our evaluative apparatus, while our core scientific contribution is a benchmark designed to evaluate the physician AI’s joint relational competence and diagnostic robustness in stigma-aware clinical consultations.
By decomposing who is evaluated in what scene and what capabilities are under test, our work studies a research question explicitly distinct from prior works rather than overgeneralizing them:
- PWP primarily evaluates patient simulators in personality-conditioned clinical encounters anchored in real consultation transcripts, assessing simulation realism, persona fidelity, and disclosure pacing. Its downstream evaluation additionally tests clinician LLMs on identifying contextual causes and avoiding care-plan contextual errors.
- MedDialBench evaluates doctor LLMs in multi-turn diagnostic consultations with graded, case-specific patient-behavior configurations fixed within each dialogue, assessing diagnostic robustness across behavior types, severity levels, and their combinations.
- MedConceal evaluates human clinicians and clinician LLMs in consultations where visible clinical context is provided but psychosocial concerns remain hidden, assessing hidden-concern elicitation and confirmation, concern-specific intervention, and guidance toward a target care plan; differential-diagnosis accuracy is not evaluated.
- , to our knowledge, is the first benchmark that evaluates whether an AI doctor can build interpersonal trust through conversational bedside manner to elicit stigma-gated, diagnostically load-bearing clinical evidence required for accurate differential diagnosis.
Response to Weakness 1.1: Motivation, Clinical Representativeness, and Nature of Patient Interaction (1/2)
We thank the reviewer for raising these thoughtful questions regarding patient psychology and clinical representativeness. We welcome this opportunity to clarify the theoretical grounding, address the SOGI literature, and present empirical evidence demonstrating that our framework models realistic, trust-gated disclosure, rather than an artificial "adversarial extreme."
1. Demographic Identity vs. Load-Bearing Clinical Stigma
The reviewer cites empirical studies [1,2] showing that patients are often willing to report their Sexual Orientation and Gender Identity (SOGI). We fully agree with these findings; however, SOGI collection fundamentally differs from the clinical taboos benchmarked in S2InQBench:
2. Defensive Concealment vs. Adversarial Interaction
The reviewer notes that "in practice, physicians and patients work together toward the same goal" and expresses concern over "adversarial interactions." We clarify that our benchmark strictly follows this goal. As formalized in $\S$3.1, clinical consultations are modeled as an imperfect-information, general-sum game rather than an adversarial or zero-sum interaction: