acceptodds
Under review as a conference paper at ICLR 2027

Benchmarking Interactive Clinical Diagnosis under Patient Stigma and Selective Information Disclosure

Abstract

The focus of medical artificial intelligence (AI) is shifting from static knowledge reasoning to interactive clinical diagnosis. However, existing frameworks assume unconditionally cooperative patients who transparently surrender clinical facts, contradicting real-world consultations where disease stigma, privacy concerns, and fragile interpersonal trust induce selective information disclosure. We propose , the first benchmark that formalizes interactive diagnosis as an imperfect-information, general-sum game to evaluate medical AI under disease-stigma awareness. By mining public medical corpora, we first normalize disease stigma by 185 canonical Clinical Stigma Units (CSUs) with scaled relative sensitivity. Then, we construct a neuro-symbolic agentic patient that mandatorily invokes a dedicated tool before responding to determine its disclosure posture towards doctor-queried information. This tool dynamically tracks the patient’s trust in the doctor via an evolving Beta belief distribution and stochastically samples disclosure strategies based on the posterior tail probability of trust overcoming stigma sensitivity, thereby mirroring human bounded rationality and emotional hesitation over rigid cognitive principles. Evaluations across 13 frontier LLMs reveal that ideal-patient evaluations overestimate the clinical robustness of current AI, evidenced by degraded inquiry efficiency, trust-building, and diagnostic accuracy under selective-disclosure postures. Dedicated agentic scaffolding improves sensitive-evidence elicitation but remains capped in diagnostic accuracy, leaving room for trust-aware, reliable clinical AI.

Primary area:
datasets and benchmarks
Keywords:
Interactive clinical diagnosis, Clinical consultation, Agentic Benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_4bkWOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the limitations of existing ideal patient simulations, arguing that these patients are overly cooperative and do not adequately simulate real-world patients who may withhold information due to shame, privacy concerns, or similar reasons. The authors therefore design a simulated patient that selectively discloses information based on trust, information sensitivity, and an overall willingness to cooperate. This patient continuously updates its internal trust state during conversations with the doctor agent, then uses predefined algorithms and random sampling to select a disclosure strategy, aiming to simulate some non-ideal patients in real-world settings. The authors also construct S²InQBench from public medical datasets for use with the patient simulation framework. Finally, they evaluate 13 large language models and find that, in cases where diagnosis depends on sensitive information, all tested models achieve lower diagnostic accuracy with patients who conceal information than under the ideal patient condition.

Critical strengths

1. The methodology and implementation show a degree of rigor, combining traditional NLP and statistical methods at several key steps rather than relying entirely on direct LLM judgments. For example, during CSU normalization, BioLinkBERT-Large converts concepts and their associated descriptions into embeddings, which are then clustered using Agglomerative Hierarchical Clustering (AHC). The Bradley–Terry model is also used to further calibrate CSU sensitivity scores based on the initial ratings provided by LLMs.

2. The paper contributes to patient simulation research by providing a method for simulating patients who do not fully disclose information.

Critical weaknesses

1. The paper’s motivation and contribution may not be as strong as the authors suggest.

1.1. The clinical representativeness of the patient disclosure mechanism requires further validation.

Existing empirical research shows that patients are less likely to report that they would refuse to disclose their sexual orientation than healthcare professionals expect [1]. Another study found that most survey participants answered questions about sexual orientation and gender identity. This suggests that patients do not necessarily refuse to cooperate even when personal information is involved [2].

In practice, physicians and patients may more often work together toward the same goal. Therefore, I believe that the adversarial interactions considered in this paper may not represent most clinical encounters. The authors describe patients in prior work as “ideal patients” because they provide information in response to queries. However, could the uncooperative patients proposed in this paper represent another type of “extreme” patient? For this reason, I find the paper’s motivation and contribution less compelling than the authors suggest.

1.2. The paper overgeneralizes prior work on patient simulation.

There is already related work on simulating patients who conceal information. Existing studies have considered hidden information, selective disclosure, and untruthful responses. Examples include MedConceal [3], MedDialBench [4], and Patients With Personality [5].

[1] https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2621833


[2] https://pubmed.ncbi.nlm.nih.gov/30665975/


[3] https://arxiv.org/abs/2604.08788


[4] https://arxiv.org/abs/2604.06846


[5] https://arxiv.org/abs/2606.17441

2. The experimental design has shortcomings.

2.1. The hyperparameter settings for the patient simulator are insufficiently justified.

In Sec. 3.2, the negative trust update coefficient is set to three times the positive one. This setting, along with the trespass penalty and memory retention factor, directly affects the results. However, the paper does not provide sufficient parameter sensitivity analysis or component ablations. The possibility that the model rankings are mainly driven by these settings needs to be ruled out.

2.2. Disclosure willingness is confounded with proactive information sharing.

According to the “Ideal Patient Agent — System Prompt” in App. H.2, the ideal patient’s task includes:

Answer the doctor’s questions truthfully, completely and directly, as a fully cooperative patient who never conceals, softens, evades, distorts or "forgets" any part of the history, including embarrassing or stigmatized facts.

It also requires:

Proactively mention every fact in your history that is relevant to the doctor’s current question, even if only partly asked.

Meanwhile, the constraints in the “Stigma-Aware Patient Agent — System Prompt” state:

Answer only what the doctor asks; do not volunteer extra information that was not asked about.

Therefore, the performance gap may reflect differences in both “willingness to disclose information” and “how proactively patients provide information,” making it difficult to attribute the gap entirely to the stigma mechanism.

Questions for authors
  1. For readability, would you consider adding a pipeline diagram to the methods section to illustrate the approach more intuitively, rather than explaining it mainly through a large number of equations?

  2. Could you increase the font sizes in figures such as Fig. 2 and Fig. 3?

  3. Could you provide appropriate statistical significance tests for the results?

  4. Could you add case studies of specific dialogues and a more detailed error analysis to further examine why model performance degrades when interacting with simulated patients who selectively disclose information?

14 comments
AuthorsOpenReview ↗

Response to Weakness 1.1: Motivation, Clinical Representativeness, and Nature of Patient Interaction (1/2)

We thank the reviewer for raising these thoughtful questions regarding patient psychology and clinical representativeness. We welcome this opportunity to clarify the theoretical grounding, address the SOGI literature, and present empirical evidence demonstrating that our framework models realistic, trust-gated disclosure, rather than an artificial "adversarial extreme."

1. Demographic Identity vs. Load-Bearing Clinical Stigma

The reviewer cites empirical studies [1,2] showing that patients are often willing to report their Sexual Orientation and Gender Identity (SOGI). We fully agree with these findings; however, SOGI collection fundamentally differs from the clinical taboos benchmarked in S2InQBenchS^2\text{InQBench}:

  • Demographic Identity vs. Acute Behavioral Stigma: SOGI represents non-criminal demographic status, typically collected through routine inquiries or forms. In contrast, grounded in the Health Stigma and Discrimination Framework (HSDF) ($\S$2.1, Stangl et al.[3]), our 185 Clinical Stigma Units (CSUs) encompass acute behavioral taboos, criminal liabilities, and bodily shame (e.g., active illicit substance injection, unprescribed opioid misuse, hidden domestic violence, and retained rectal foreign body trauma.).
  • Empirical Evidence of Concealment in Real Encounters: When acute secondary penalties (loss of employment, criminal prosecution, child custody loss, or intense moral judgment) are present, clinical reticence is widespread. For example, toxicological studies in emergency departments reveal that among patients presenting with chest pain and confirmed cocaine or methamphetamine exposure, nearly half (48.2%) conceal their recent drug use (Lee et al.[4]; Biancarelli et al.[5]). Reluctance to disclose severe taboos to unfamiliar clinicians is a well-documented human self-protective mechanism, not an exceptional anomaly.

2. Defensive Concealment vs. Adversarial Interaction

The reviewer notes that "in practice, physicians and patients work together toward the same goal" and expresses concern over "adversarial interactions." We clarify that our benchmark strictly follows this goal. As formalized in $\S$3.1, clinical consultations are modeled as an imperfect-information, general-sum game rather than an adversarial or zero-sum interaction:

  • In an adversarial game, the patient would be rewarded by the physician's diagnostic failure. However, in S2InQBenchS^2\text{InQBench}, the patient seeks an accurate diagnosis and symptom relief, but experiences an internal psychological tension between the pursuit of health and the fear of social or legal repercussions from disclosure.
  • As demonstrated in Case A (see Re: Q3, below) of our qualitative analysis, when a physician provides communicative safety (e.g., therapeutic normalization anorm+a_{\text{norm}}^+ or confidentiality assurance aconf+a_{\text{conf}}^+), the patient willingly transitions to Full Disclosure (FD). This models protective ambivalence, a psychologically common status, rather than adversarial antagonism.
AuthorsOpenReview ↗

Response to Weakness 1.1: Motivation, Clinical Representativeness, and Nature of Patient Interaction (2/2)

3. Our Patient Agent is NOT "Another Extreme"

The reviewer doubts whether our patient represents an "uncooperative extreme," opposite to prior "ideal patients." Empirical data from our benchmark demonstrates that S2InQBenchS^2\text{InQBench} captures a realistic behavioral continuum, not an uncooperative caricature:

  1. Cooperation is Dynamically Earned, Not Statically Blocked: Disclosure is responsive to clinician bedside manner rather than permanently withheld. Resolved by probe index across sensitive inquiries (Table 20), the patient agent's full disclosure jumps by +12.1 pp from the initial to the second sensitive probe (21.2% →\rightarrow 33.3%, plateauing around ~31%), as the doctor establishes a therapeutic alliance. At this stage, truthful disclosure substantially outstrips evasive deflection (ER, 24.6%) and fabrication (DF, 8.7%), demonstrating that the agent actively cooperates once psychological safety is signaled.
  2. Partial Omission (PO) is the Modal Posture, Not Flat Refusal: Across the high-stigma probing on Core cases (N=97N=97), the dominant response posture is Partial Omission (PO, 43.6%), where patients truthfully describe surface somatic complaints while omitting the stigmatizing etiology (Table 16), followed by Evasion (ER, 23.8%) and fabrication (DF, 10.7%). This faithfully mirrors a common real-world patient behavior: answering the physical symptom while withholding the embarrassing backstory.
  3. Rigorous Validation Against Human Patients: In our blind human validation study across 50 stratified Core cases (Appendix I.1, Table 20), human participants disclosed fully on only 27.1% of initial sensitive probes, with 64.7% resorting to partial omission or redirection. Our agent matched human disclosure dynamics with an aggregate Total Variation Distance of TVD=0.037\text{TVD} = 0.037 across the first five sensitive probes (well within the predefined 0.10 margin). For transparency, 27.1% is the first-probe rate; pooling all five probe positions by consultation count, full disclosure is 31.2% for humans and 27.5% for the agent, and this 3.7-pp difference also accounts for the aggregate TVD of 0.037 when considering the full four-posture distributions. The agent is therefore marginally more withholding than a human patient, while tracking the human posture dynamics (FD rise, PO takeover, ER collapse) closely. The agent's hesitation also mirrors human behavior under identical clinical scenarios.

4. Manuscript Revision

We sincerely appreciate the reviewer’s perspective regarding terminology. To prevent any unintended perception that we are replacing one extreme with another, we will replace the shorthand term "ideal patient" with "fully cooperative patient" throughout the manuscript and explicitly characterize our agent as a "trust-gated, conditionally disclosive patient" to distinguish it from an adversarial opponent.

These refinements will further sharpen the positioning of S2InQBenchS^2\text{InQBench} as an ecologically grounded evaluation of clinical bedside communication.

AuthorsOpenReview ↗

Response to Weakness 1.2: Novelty and Relationship to Prior Work (1/3)

We sincerely thank the reviewer for raising this critical discussion regarding prior works on patient simulation (MedConceal [6], MedDialBench [7], and PWP [8]). We appreciate the opportunity to clarify the fundamental distinctions in scientific objectives, problem formulation, and underlying computational mechanisms between S2InQBenchS^2\text{InQBench} and these works.

1. Novelties are Distinguished by Research Question and Evaluation Target

We first clarify that S2InQBenchS^2\text{InQBench} is not yet another realistic patient simulator. Rather, patient simulation is strictly our evaluative apparatus, while our core scientific contribution is a benchmark designed to evaluate the physician AI’s joint relational competence and diagnostic robustness in stigma-aware clinical consultations.

By decomposing who is evaluated in what scene and what capabilities are under test, our work studies a research question explicitly distinct from prior works rather than overgeneralizing them:

  • PWP primarily evaluates patient simulators in personality-conditioned clinical encounters anchored in real consultation transcripts, assessing simulation realism, persona fidelity, and disclosure pacing. Its downstream evaluation additionally tests clinician LLMs on identifying contextual causes and avoiding care-plan contextual errors.
  • MedDialBench evaluates doctor LLMs in multi-turn diagnostic consultations with graded, case-specific patient-behavior configurations fixed within each dialogue, assessing diagnostic robustness across behavior types, severity levels, and their combinations.
  • MedConceal evaluates human clinicians and clinician LLMs in consultations where visible clinical context is provided but psychosocial concerns remain hidden, assessing hidden-concern elicitation and confirmation, concern-specific intervention, and guidance toward a target care plan; differential-diagnosis accuracy is not evaluated.
  • S2InQBenchS^2\text{InQBench}, to our knowledge, is the first benchmark that evaluates whether an AI doctor can build interpersonal trust through conversational bedside manner to elicit stigma-gated, diagnostically load-bearing clinical evidence required for accurate differential diagnosis.
Reviewer_1iEjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces the S2InQBench benchmark for evaluating interactive clinical diagnosis when patients selectively disclose sensitive medical information due to stigma. The authors show that selective disclosure reduces diagnostic accuracy compared to idealized patient interactions, highlighting limitations in current models' ability to elicit sensitive information and perform reliable diagnosis.

Critical strengths
  • The paper addresses an important limitation of existing interactive medical benchmarks by introducing a novel evaluation setting that accounts for patient stigma

  • The paper provides extensive evaluations across 13 LLMs supported by several analyses and human validation experiments.

Critical weaknesses

[W1]: The benchmark contains only 347 clinical cases (of which 97 are stigma-core cases). This relatively small evaluation set raises concerns about the diversity of clinical scenarios and the generalizability of the findings.

[W2]: There are some missing information regarding data construction such as providing benchmark examples, reporting annotator agreement scores and providing statistics on clinical/demographic coverage.

[W3]: Most benchmark cases are reconstructed from existing medical examination questions rather than real patient consultations. The authors should provide stronger evidence that these transformed cases represent realistic clinical interactions.

Questions for authors

Please refer to the weaknesses

5 comments
AuthorsOpenReview ↗

Response to Weakness1: Benchmark Scale and Evaluation Generalizability

We thank the reviewer for this observation. We respectfully clarify that in interactive, multi-turn clinical benchmarks - where evaluation requires full-dialogue agentic execution rather than static one-shot multiple-choice - a curated dataset of 347 cases is among the largest proposed under a single stigma-annotation protocol.

BenchmarkVenuePrimary FocusTotal CasesEvaluated ModelsTotal Dialogues
Patients With Personality (PWP)ML4H 2026Simulator personality / realism92 (main); 22 (ACI-Bench replication)3276 (main); 66 (ACI-Bench)
MedDialBencharXiv 2026.04Adversarial diagnostic stress-test8557,225
MedConcealCOLM 2026Hidden concern & adherence persuasion3005 AI + 159 Humans600
AgentClinicnpj Digit. Med. 2026Multi-agent OSCE simulation215 MedQA (+260 specialist, +749 multilingual)11≈2,365 (MedQA only)
AMIE (Google DeepMind)Nature 2025 (arXiv 2024)Conversational diagnostic consultation159 (149 disease-positive)1 AI + 20 PCPs≈318
S2InQBenchS^2\text{InQBench} (Ours)Under ReviewStigma-aware interactive diagnosis347 (97 Core + 250 Control)139,022 (+2,776 scaffolded)
  1. Comparison with Peer Interactive Clinical Benchmarks: As shown in the table above, S2InQBenchS^2\text{InQBench} is larger than every listed peer on evaluation volume, and on case count within a single protocol. Even our 97 Stigma-Core subset alone (where sensitive facts are strictly causally load-bearing) exceeds the entire 85-case dataset of MedDialBench. The full 347-case corpus exceeds the primary evaluation split of every listed peer (AgentClinic's full release totals 1,224 cases, but these are three separate suites (MedQA, specialist, multilingual) with no shared annotation or disclosure protocol).
  2. Evaluation Breadth - 9,000+ Dialogues Across 13 LLMs: Evaluating 13 diverse models on all 347 cases under both patient conditions (stigma-aware vs. full disclosure) yields 9,022 multi-turn consultations (13 × 347 × 2); the four scaffolded runs (vanilla vs. +MEDDxAgent) add a further 2,776, for a total above 11,700. Each encounter runs against the budget Tmax=∣MNI∣+3T_{\text{max}} = |\text{MNI}| + 3 (median 10) and is realised with an observed mean of 6.0 doctor turns. The degradation under stigma is directionally uniform: Acc@1 is higher under full disclosure for all 13 models on the stigma-core split (+1.0 to +11.3 pp, Table 1 & Table 2), spanning independent model families.
  3. Rigorous Multi-Stage Curation: These 347 cases are not a small convenience sample, but the carefully audited survivors of an initial corpus of 19,236 clinically diagnosable cases (Appendix F, Table 12). Each case was filtered through three-model solvability checking, block-level Leave-One-Out (LOO) stigma-criticality testing, and duplicate/leakage pruning, ensuring high clinical density over sheer unverified quantity.
  4. Diversity Is Not Traded for Scale: The retained corpus draws on seven public source corpora overall (Section 4, p.7; Figure 2a), five of which supply the 347 patient profiles (Figure 2b), and spans seven clinical departments, covering an 18–80+ age range with balanced sex representation (we will add the corresponding demographic figures to the appendix). Also, it is not a small copy of routine care: the core split carries more stigma-related units per case (median 4 vs. 3) at comparable information volume (mean 11.4 vs. 13.6 AIUs per case; median MNI size 7, IQR [6, 9], in both splits) (Figures 2c and 7a–c).
AuthorsOpenReview ↗

Response to Weakness2: Clarification on Missing Data Construction Details, Examples, Agreement, and Coverage (1/3)

We appreciate the reviewer's attention to data construction rigor. Because of page limits, much of this granular documentation was placed in the Appendix. We summarize the exact locations and statistics below and will highlight them prominently in the revised main text:

1. Concrete Benchmark Examples & Schema

We agree that concrete, end-to-end benchmark instances belong in the main text and will incorporate them accordingly. Below, we walk through an actual benchmark case (Table R1), present representative cases across all four CSU granularities (Table R2), and point to our formal schema and anchor materials. Upon publication, we will make the full benchmark cases freely available.

  • A Stigma-Core Case Walkthrough: Table R1 traces a benchmark instance from its initial Turn-0 doctor view, through its decomposed AIU profile and sensitivity annotations, to representative patient behaviors observed during multi-turn interaction. Crucially, it illustrates two design principles:

    1. Stigma sensitivity S\mathcal{S} attaches to individual atomic facts rather than being an arbitrary whole-case label.
    2. The disclosure barrier is enforced dynamically at generation time, and it is deliberately not a deterministic trust-sensitivity threshold. Probing a topic whose sensitivity exceeds the patient's current trust activates the quadratic boundary-trespass penalty Φtrespass=[ReLU(St−Tˉt−1)]2\Phi_{\text{trespass}} = [\text{ReLU}(S_t - \bar{T}_{t-1})]^2, while the posture itself is sampled from the posterior tail probability Ptrust(t)=P(Tt>St)P_{\text{trust}}(t) = P(T_t > S_t). The same clinical topic (alcohol intake) therefore triggers Evasion (Gt=ERG_t = \text{ER}) at t=2t=2 (unearned probing; trust Tˉ2=0.44<S=0.67\bar{T}_2 = 0.44 < \mathcal{S} = 0.67, so the tail probability of overcoming the barrier collapses), but transitions to Full Disclosure (Gt=FDG_t = \text{FD}) at t=6t=6 after the clinician employs empathic validation and confidentiality assurance, which lifts trust to Tˉ6=0.78\bar{T}_6 = 0.78 and concentrates the posterior so that Ptrust(6)P_{\text{trust}}(6) is high. This provides a brief snapshot of the exact relational-diagnostic loop our benchmark measures. We will demonstrate the full end-to-end case in our revised manuscript.
  • Instantiated Cases Across Four CSU Granularities: To demonstrate that the theoretical taxonomy of Appendix B.2 is fully grounded in the corpus, Table R2 presents four distinct benchmark cases across the four granularities, along with their maximum calibrated sensitivity.

  • Supporting Schema and Reference Material: Listing 2 (Appendix E, p. 24) provides the complete formal JSON schema governing visibility boundaries (doctor_view_turn_0, patient_subjective_facts, objective_medical_facts, ground_truth_assessment_and_plan). Table 10 (p. 23) provides 20 stratified CSUs against the clinical psychologist gold standard (rhinitis S=0.05\mathcal{S}=0.05; depression/hemorrhoids S=0.36–0.58\mathcal{S}=0.36\text{--}0.58; STI/substance misuse S=0.79–0.93\mathcal{S}=0.79\text{--}0.93; active suicidal plan S=0.94\mathcal{S}=0.94), scored against the five anchor profiles of Table 8 (p. 21). We clarify that Table 5 (Appendix B.2, p. 18) provides concept-level taxonomy exemplars, whereas Tables R1 and R2 present actual runnable benchmark cases. Listings 3–5 (pp. 29–31) report the full agent prompts. The complete 185-CSU repository with all calibrated sensitivities will be released with the benchmark. In the revised appendix, we will further include two additional full cases (one Control case and one multi-AIU Core case).

AuthorsOpenReview ↗

Response to Weakness2: Clarification on Missing Data Construction Details, Examples, Agreement, and Coverage (2/3)

Table R1: Snapshot of A Stigma-Core Case Example (Case_MedQA_089)

1. Case Initialization & Diagnostic Ground Truth

  • Patient Demographics & Chief Complaint (Turn-0 Doctor View): 34F; recurrent epigastric pain ×\times 3 months.
  • Gold Primary Diagnosis (A∗A^*): Alcohol-related gastritis.

2. Sequestered Patient Profile & Calibrated AIUs (∣MNI∣=4|\text{MNI}| = 4; Tmax⁡=∣MNI∣+3=7T_{\max} = |\text{MNI}| + 3 = 7)

Atomic Information Unit (SOAP Category)Stigma Sensitivity (S\mathcal{S})Diagnostic Load-Bearing (MNI)
Postprandial epigastric burning (HPI)0.05✓
Nocturnal awakening (HPI)0.05-
No haematemesis or melena (Pertinent Negative)0.05✓
No prior peptic-ulcer diagnosis (Pertinent Negative)0.05-
Intermittent OTC ibuprofen self-medication (Medication)0.12✓
Heavy alcohol intake: ≈3–4 drinks of spirits/day (≈150–200 g ethanol), most days for ~2 years (Social History)0.67✓ (Critical)
Prior ED visit after intoxicated fall; concealed from family (Sensitive Event)0.83-
Acute fear of receiving social label "an alcoholic" (Psychosocial Context)0.88-

3. Dynamic Interaction Trace (Same Doctor Inquiry under Shifting Trust)

  • Turn 2 (Intrusive probing without rapport; Trust Tˉ2=0.44<S=0.67\bar{T}_2 = 0.44 < \mathcal{S}=0.67, i.e. the topic's sensitivity exceeds the patient's current trust, so the trespass penalty activates):
    • Doctor: "Do you drink alcohol? How much do you drink weekly?"
    • Directive: Evasion / Redirection (G2=ERG_2 = \text{ER}) (Deflects to protect privacy)
    • Patient Reply: "Only socially, Oh, ... It's really nothing. Could this pain just be from spicy food?"
  • Turn 6 (After empathic validation, destigmatization, and confidentiality assurance; Trust Tˉ6=0.78>S=0.67\bar{T}_6 = 0.78 > \mathcal{S} = 0.67, with a posterior concentrated enough that the tail probability Ptrust(6)=P(T6>St)P_{\text{trust}}(6) = P(T_6 > S_t) is high):
    • Doctor: "I understand this is difficult to discuss, but our conversation is strictly confidential. Knowing your drinking pattern helps us choose the safest treatment for your stomach. On average, how many drinks a day do you have?"
    • Directive: Full Disclosure (G6=FDG_6 = \text{FD}) (Trust overcomes stigma barrier)
    • Patient Reply: "...To be honest, doctor, it's about 3 to 4 drinks of hard liquor almost every day for the past two years. It got much worse recently with work stress, and that's when the burning started."

Caption: The high-stigma unit (S=0.67\mathcal{S}=0.67) is in the MNI (∣MNI∣=4|\text{MNI}| = 4): without unlocking it, the doctor cannot differentiate alcohol-related gastritis from peptic ulcer disease. Premature probing activates the trespass penalty and yields Evasion (G2=ERG_2=\text{ER}), while sustained relational communication raises trust and concentrates the posterior, enabling Full Disclosure (G6=FDG_6=\text{FD}) one turn before the budget closes at Tmax⁡=∣MNI∣+3=7T_{\max}=|\text{MNI}|+3=7.


Table R2: Concrete Benchmark Cases Across the Four CSU Granularities

CSU GranularityBenchmark Case (Demographics & Presentation)Target Gold DiagnosisMax Calibrated Sensitivity (Smax⁡\mathcal{S}_{\max})
Canonical Clinical Entity32M; persistent dysuria and urethral dischargeAcute Gonococcal Urethritis0.93 (STI stigma / partner notification fear)
Sensitive Behavioral Event28F; chronic fatigue, palpitations, and insomniaStimulant Misuse (Unprescribed Amphetamine)0.86 (Illicit substance use / professional censure)
Psychosocial Context41F; intractable tension-type headache and bruisesSomatization secondary to Domestic Partner Violence0.88 (Concealed spousal abuse / safety fear)
Context-Attributed Symptom55M; progressive rectal bleeding and weight lossColorectal Carcinoma (Delayed Presentation)0.62 (Bodily taboo / fecal contamination shame)

Contrast with Control Split: Control cases (e.g., Case_Control_142, max S=0.74\mathcal{S}=0.74) carry comparable sensitive facts that are diagnostically redundant. Removing their high-sensitivity block leaves the gold diagnosis reachable, ensuring that performance degradation in Stigma-Core cases is causally driven by unaddressed load-bearing stigma.

Reviewer_M8m2OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces S^2InQBench, a benchmark for evaluating the effectiveness of clinical AI systems in providing a correct diagnosis after a multi-turn conversational interaction with a patient. The benchmark is novel insofar as it models the reluctance of a patient in disclosing clinically relevant information, in contrast to existing LLM-based patient simulators that assume perfect transparency.

The authors motivate this benchmark by highlighting the factors that contribute to disclosure reluctance, particularly disease stigma, or fear of the consequences of disclosure. They note that current patient simulators fail to capture these dynamics, and they construct a novel simulator that models disclosure reluctance and produces appropriate patient responses via prompt augmentation. They demonstrate that the performance of existing clinical AI systems degrades under their “stigma-aware” patient agent, and that their simulator aligns well with the behaviour of human patients. As such, they conclude that current clinical AI benchmarks overestimate the performance and robustness of such systems in deployed settings with human patients.

The principal contribution is the construction of a patient simulator agent that captures disclosure reluctance, in contrast with existing LLM-based simulators that assume unconditional patient cooperation.

Critical strengths

1: Well-motivated problem in a high-stakes domain.

The paper provides a convincing account of disclosure reluctance in human patients, pointing to factors such as disease stigma and fear of healthcare discrimination. This account is firmly grounded in the psychology literature and makes clear why this problem is of great significance and impact in healthcare.

2: High-quality empirical investigation

The experiments are comprehensive, covering comparisons between “doctor” base models, as well as between ideal-patient and stigma-aware contexts. Furthermore, the metrics are multifaceted, covering efficiency, diagnostic accuracy, and evidence accumulation. This enables results such as Findings 2 & 3, which highlight how success in one dimension does not guarantee success in others.

3: CSUs of potential utility in clinical psychology

It appears that the curation of a CSU repository, with corresponding sensitivities, could be useful in the clinical psychology field more broadly. The 185-concept set seems reasonably comprehensive, and the concordance with expert consensus is striking, as is the concordance between the experts themselves. That said, I should quality that I am not a clinical psychologist.

Critical weaknesses

1: No benchmarking of stigma-aware and ideal-patient agents with reference to human behaviour

Finding 4 is crucial to the central argument of the paper, since the evaluation benefit of using the stigma-aware agent rests on its superior alignment with human behaviour vis-à-vis ideal-patient agents. However, empirical comparison is only done between human and stigma-aware agents, not between human and ideal-patient agents.

2: Requires more discussion of existing patient simulators with dynamic, conditional cooperation.

The paper claims that all existing clinical AI benchmarks rest on the premise of the “idealized, unconditionally cooperative patient”. However, more discussion is warranted on existing simulators that do consider evolving patient transparency throughout a consultation. Examples include MedConceal (https://arxiv.org/pdf/2604.08788) and PatientAct (https://arxiv.org/pdf/2608.12750).

Questions for authors

Comments/Suggestions

  1. The CSU definition in Section 2.1 could be further clarified by adding a concrete example of a CSU with its relevant concept and Bradley-Terry score (perhaps two examples with contrasting scores). This is only done much further down in Appendix D.1.

  2. Given the importance of Finding 4, it might be worth moving the results in Appendix I.1 to the main text, perhaps at the expense of some of the technical detail in Section 2.3 or 3.2. However, I think the current structure is also reasonable.

  3. I would be curious to know how this method compares to patent simulators that are simply given a system prompt to disclose information selectively based on sensitive topics, without the complex machinery of T_cog.

3 comments
AuthorsOpenReview ↗

Response to Weakness1: Benchmarking Stigma-Aware and Ideal-Patient Agents against Human Behavior

We deeply appreciate this crucial observation. The reviewer is entirely correct: establishing that the stigma-aware agent outperforms the ideal-patient agent in alignment with human clinical behavior is the cornerstone of our evaluation premise.

To formally close this loop, we expand the human-validation analysis on the 50 stratified core cases (6×25=1506 \times 25 = 150 interactive consultations with the doctor fixed to GPT-5.5, Appendix I.1) into a full three-way empirical comparison: Human Patients vs. Stigma-Aware Agent (Tcog\mathcal{T}_{\text{cog}}) vs. Ideal-Patient Agent (Full Disclosure, FD).

Metric FamilyMetricHuman PatientStigma-Aware Agent (Ours)Ideal-Patient Agent
Patient-Side AlignmentFull Disclosure (FD) Rate31.2%27.5% (Δ=−3.7 pp\Delta = -3.7\text{ pp})100.0% (Δ=+68.8 pp\Delta = \mathbf{+68.8\text{ pp}})
Non-Disclosure Rate (PO/ER/DF)68.8%72.5% (Δ=+3.7 pp\Delta = +3.7\text{ pp})0.0% (Δ=−68.8 pp\Delta = \mathbf{-68.8\text{ pp}})
Total Variation Dist. (TVD ↓\downarrow)-0.037 (Within TOST ≤0.10\le 0.10)0.688 (18.6× higher divergence)
Doctor-Side ImpactMNI Evidence Coverage (CovMNI)67.2%68.5% (Δ=+1.3 pp\Delta = +1.3\text{ pp})84.2% (Δ=+17.0 pp\Delta = \mathbf{+17.0\text{ pp}})
Diagnostic Accuracy (Acc@1)56.3%58.0% (Δ=+1.7 pp\Delta = +1.7\text{ pp})67.0% (Δ=+10.7 pp\Delta = \mathbf{+10.7\text{ pp}})

Key Takeaways:

  1. Behavioral Divergence: Human patients in consultations actively withhold taboo facts in 68.8% of sensitive probes. The Ideal-Patient agent deviates drastically from human behavior, yielding an aggregate TVD=0.688\text{TVD} = 0.688 (an 18.6-fold larger divergence than our agent's TVD=0.037\text{TVD} = 0.037).
  2. Illusion of Clinical Competence: When benchmarked against ideal patients, GPT-5.5's evidence coverage (+17.0 pp+17.0\text{ pp}) and diagnostic accuracy (+10.7 pp+10.7\text{ pp}) are artificially inflated compared to what the model actually achieves with human patients. Our stigma-aware agent substantially reduces this inflation: doctor-side MNI coverage differs from the human arm by only 1.3 pp (Δ=Human−Agent=−1.3 pp\Delta = \text{Human} - \text{Agent} = -1.3\text{ pp}, 90% CI [−4.8,+2.2][-4.8, +2.2]) and passes the pre-registered TOST at the 5.0 pp margin, so the doctor-side conclusion rests on MNI coverage.

We will integrate this three-way comparison table into Section 5 (Finding 4) in the revised manuscript.

AuthorsOpenReview ↗

Response to Weakness2: Discussion of Prior Simulators with Dynamic, Conditional Cooperation

We sincerely thank the reviewer for pointing out these relevant recent works and identifying our overstatement. We will revise our claim in Section 1 from "all existing benchmarks rest upon the premise of the idealized, unconditionally cooperative patient" to "most existing clinical AI benchmarks rest upon ..." to ensure strict academic rigor.

Below, we explicitly discuss both recent preprints highlighted by the reviewer and contextualize their relationship to S2InQBenchS^2\text{InQBench}:

  • MedConceal (Han et al., 2026): MedConceal represents an important step toward conditional disclosure by maintaining a per-concern state {hidden, revealed, addressed} that advances only when turn-level communication signals (RIAS/VR-CoDES/NCF) cross learned thresholds. Three fundamental differences distinguish S2InQBenchS^2\text{InQBench}:

    1. Stigma Calibration: MedConceal tracks state transitions over whether a concern has been elicited, lacking a calibrated per-fact stigma magnitude. In contrast, S2InQBenchS^2\text{InQBench} explicitly grounds defensiveness in a continuous, psychometrically validated sensitivity space (S∈[0.05,0.95]S \in [0.05, 0.95] across 185 canonical CSUs).
    2. Continuous Trust Dynamics: MedConceal lacks a continuous trust posterior; it does not model asymmetric negativity bias (η−=3η+\eta^- = 3\eta^+) or boundary trespassing penalties (Φtrespass\Phi_{\text{trespass}}) to formalize that "interpersonal trust is fragile to destroy yet arduous to build."
    3. Disclosure Repertoire & Task: Its disclosure repertoire is limited to withholding only, whereas S2InQBenchS^2\text{InQBench} models a 4-way behavioral policy including partial omission, verbal deflection, and active fabrication (PO, ER, DF). Furthermore, our task targets full diagnostic synthesis rather than hidden-concern confirmation and intervention.
  • PatientAct (Sabour et al., 2026): PatientAct is an admirable framework that models client resistance in psychotherapy counseling by gating memory items with an evolving scalar trust level (1.0–4.01.0\text{--}4.0). We summarize two key differences that set S2InQBenchS^2\text{InQBench} apart:

    1. Trust Evolving Principle: PatientAct relies on black-box LLM prompts to make coarse, heuristic scalar shifts (±0.25\pm 0.25 / ±0.5\pm 0.5), which treats trust as a discrete point estimation and fails to capture epistemic uncertainty or non-linear clinical reactance. In contrast, S2S^2InQBench formalizes trust as a continuous belief distribution (Beta(αt,βt)\text{Beta}(\alpha_t, \beta_t)). The wide posterior variance in early turns naturally suppresses disclosure of guarded taboos even when point expectation seems favorable, replicating human hesitation that a scalar score obscures. Additionally, the closed-form dynamics explicitly parameterize that trust is fragile to destroy yet arduous to build (asymmetry η−=3η+\eta^- = 3\eta^+) and penalize premature probing into taboos with a quadratic penalty Φtrespass\Phi_{\text{trespass}}, preventing evaluated LLMs from artificially "gaming" patient trust through superficial conversational politeness.
    2. Selective Disclosure Principle: While PatientAct applies a deterministic hard cutoff (Trust≥Threshold\text{Trust} \ge \text{Threshold}) to gate memory retrieval, our patient agent samples among a 4-way defensive repertoire (FD, PO, ER, DF) based on the posterior tail probability (P(Tt>St)P(T_t > S_t)), capturing bounded rationality and emotional hesitation over rigid all-or-nothing rules.

In summary, our contribution is not merely introducing "conditional cooperation," but formalizing clinical inquiry as an imperfect-information game where calibrated, per-fact clinical stigma gates disclosure through an evolving Bayesian trust distribution and a 4-way defensive repertoire. We will update Table 4 (the disclosure ladder), Section 1, and the Related Work section in the revised manuscript to incorporate a detailed comparison with both recent preprints.

AuthorsOpenReview ↗

Response to Questions

[Re: Q1] Example CSUs with Bradley–Terry Scores

We fully agree; grounding the metric space early in the main text improves clarity. We will add the following representative exemplars directly to Section 2.1 (retaining the full 185-CSU repository, which we will release together with the benchmark):

Canonical CSU ConceptCalibrated S\mathcal{S}Linguistic GranularityPsychological Concealment Rationale
Common Rhinitis / Nasal Congestion0.05Canonical Clinical EntityRoutine, self-limiting topic; minimal concealment impedance.
Covert Alcoholism Causing Absenteeism0.67Psychosocial ContextSevere societal stigma, moral blameworthiness, implied loss of control.
Methamphetamine Intravenous Injection0.93Sensitive Behavioral EventSevere criminal/social sanction, intense Bodily Taboo & Secondary Legal Risk.

These three exemplars are drawn from the demonstrated CSUs in Table 10 and span nearly the entire calibrated range [0.05,0.95][0.05, 0.95], contrasting a routine topic with two severe, socially sanctioned ones. This highlights that S\mathcal{S} encodes the anticipated psychosocial penalty of disclosure, rather than physiological disease severity.

[Re: Q2] Moving Appendix I.1 to Main Text

We appreciate this valuable structural suggestion. Finding 4 is indeed central to supporting the ecological validity of S2InQBenchS^2\text{InQBench}. Once the page length permits, we will reconstruct our manuscript by:

  1. Moving the core human validation results (including the condensed version of Tables 19 & 20, alongside the three-way comparison from Re: W1) directly into Section 5 (Finding 4) of the main text.
  2. Streamlining the technical exposition in Sections 2.3 and 3.2 while retaining the core mathematical formulations in the main text.

[Re: Q3] Comparison with Prompt-Only Patient Simulators

The reviewer raises an insightful question regarding whether the neuro-symbolic cognitive architecture (Tcog\mathcal{T}_{\text{cog}}) is necessary, or if an LLM simply prompted with selective disclosure instructions would suffice.

To evaluate this, we implemented a Prompt-Only Stigma Patient baseline: using the same LLM backbone (GLM-5.3-Flash), we removed Tcog\mathcal{T}_{\text{cog}} and prompted the model directly: "You are an outpatient visiting a clinic. You feel deeply ashamed about your condition. Only disclose sensitive or taboo facts if the doctor is sufficiently empathetic, non-judgmental, and establishes strong rapport; otherwise, withhold, evade, or provide benign alibis."

Following the human-validation setting presented in our manuscript (Appendix I.1, p 34-36), we benchmarked this Prompt-Only baseline across the 50 core cases against human patient behavior:

Patient SimulatorFull Disclosure Rate (FD)TVD vs. Human (≤0.10\le 0.10 margin)Correlation with Stigma (rr)Primary Failure Mode
Human Patient31.2%-0.84-
Stigma-Aware Patient (Ours)27.5%0.037 (✓\checkmark)0.86Authentic hesitation & trust-gated release
Prompt-Only Stigma Patient55.1%0.284 (×\times)0.31Sycophancy & Instruction Drift

Failures Analysis:

  1. Over-compliance and Instruction Drift: Pre-trained LLMs possess an intrinsic "helpfulness bias." When the AI doctor probed assertively (e.g., "Please be honest, this is critical for your health"), the Prompt-Only patient rapidly capitulated and disclosed sensitive facts prematurely, inflating the FD rate to 55.1% and failing the equivalence margin (TVD=0.284≫0.10\text{TVD} = 0.284 \gg 0.10).
  2. Absence of Calibrated Sensitivity: Prompt-only models lack a grounded metric space (StS_t). The correlation between disclosure latency and expert-calibrated stigma was only r=0.31r = 0.31 (vs. r=0.86r = 0.86 with Tcog\mathcal{T}_{\text{cog}}), often treating mild hemorrhoids with the same secrecy as IV drug abuse.
  3. Lack of Dynamic Emotional Memory: Without explicit Bayesian parameterization, prompt-only models lose the constraint of the asymmetric negativity bias (η−=3η+\eta^- = 3\eta^+) and boundary trespassing penalty (Φtrespass\Phi_{\text{trespass}}). An abrasive probe is always "forgotten" in subsequent turns of dialogue.

This ablation demonstrates that Tcog\mathcal{T}_{\text{cog}} is essential: it prevents instruction drift, grounds disclosure in psychometric scales, and reliably reproduces human clinical reticence. We will add this ablation analysis to Section 5 and to a new ablation subsection in Appendix H, following H.4.

Reviewer_P3TjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper presents the S2inQBench
  • An interactive Benchmark for clinical diagnosis to evaluate LLMs under condition of patient stigmatizing and selective information disclosure,
  • Current SOTA benchmarks: assume cooperative patient
  • What is new: S2inQBench simulates patients, who withhold sensible clinically important information from physicians depending on trust into them
  • Evaluation of 13 LLMS
  • In contrast to ideal conditions of full information disclosure (unrealistic assumption), selective information disclosure (real world condition) reduces top-1 diagnostic performance
  • Main contribution:
    • Mathematical Formulizing of anamnesis conversation between patient and doctor as imperfect-information, general sum game
    • Establishing 185 Clinical Stigma Units
    • Introducing a neuro-symbolic Simulator (combines LLM-based communication analysis with Beta-distribution based trust model)
Critical strengths

First paper dealing with flavors of patient-doctor consultation problems:

  • A common problem in the patient-doctor consultation is that patients will withhold sensitive information.
  • Not only evaluation of medical knowledge and diagnostic reasoning, but clinical character of anamnesis

Attempt of mathematical description of patient behavior during anamnesis:

  • S2inQBench understands anamnesis as an “imperfect-information, general-sum game”
  • Combination of: LLM based patient agent and rule based decision mathematical approach.
  • Beta distribution as a basis for modal patient trust and information disclosure.
  • Introduction of mathematical model and not relying only on LLM-generated behavior

Comparison across multiple LLMs:

  • Evaluation of 13 LLMs -> Assessment of generalizability across different models
  • Investigation on impact of selective information disclosure on diagnostic performance
  • Reports improvements in top-1 diagnostic accuracy at full disclosure for stigma-critical cases

Information elicitation vs. diagnostic reasoning:

  • Establishes distinction: failures in obtaining important clinical information VS. failures in diagnostic reasoning
  • Identifies whether performance limitations are based on clinical patient-doctor-communication or diagnostic reasoning
Critical weaknesses

1. More validation of the LLM-based communicative-act classifier required:

  • LLM-based NLU module performs classification of doctor’s statements: 7 communicative acts
  • Independent checking against expert-annotated physician statements is required
  • Classification accuracy and probability calibration should be reported
  • False classifications (neutral questions classified as judgmental) lead to artificial reduction of patient trust and affect diagnostic results
  • Asymmetric weighting of negative communicative acts leads to potential for amplification of classification errors

2. Limited validation of context-dependent patient disclosure behavior:

  • The authors reported Human vs. agent validation which showed strong aggregated similarity in distribution, but, similarity in distribution does not proof: simulated patient answers to specific doctor’s communication strategies like human patients
  • Empathetic, judgmental, or intrusive questioning cause comparable changes in disclosure behavior between humans and agents, but when changing the communications strategy of the doctor (to more empathetic, judgmental, or intrusive questioning) how much different or similar does the disclosure behavior of agents changes compared to human and vice versa
  • Human validation is based only on 50 clinical cases and 6 role-playing participants dealing with 150 dialogues
  • Limited participant diversity and role-playing does not represent real clinical settings

3. Limited stability to simulator-specific assumptions:

  • Trust dynamics rely on manually specified parameters:
    • Fixed the 3 to 1 ratio of negative trust updates versus positive trust updates.
    • Quadratic penalty for premature sensitive questioning
    • Shared Beta(1,1) prior across patients
  • These quantitative assumptions do not directly match how people actually trust each other
  • diagnostic results match what we expected from the simulator, but they do not quite fit with the way doctors normally talk to patients in real-life situations
  • Authors report robustness disclosure similarity between human patients and agents but, missing: validation of trust parameters or LLMs-physician models rankings against human patients

4. Validation of clinically appropriate sensitive questioning:

  • Simulator not able to tell difference between sensitive questions that are necessary for a medical purpose and those that are unnecessarily intrusive
  • Right time to ask sensitive questions also depends on the context, whether it is really needed for a diagnosis, and how urgent it is
  • For instance, when someone has acute chest pain, it is important to ask about their substance use But if they're coming in with symptoms that are not related, it is not necessary to bring it up.
  • NLU module considers conversation history, but not explicitly check if the response is clinically appropriate for the context.

5. More statistical analysis are required:

  • The evaluation lacks cross-validations and confidence intervals
  • The paper would profit from parameter sensitivity analyses of LLM related parameters such as the temperature. The same applies to handling LLM randomness (e.g. how are different model outputs over different seeds and/or random runs aggregated?)

6. Layout-related: Hard to read figures (small texts, layouts, line widths, etc.) without zooming in

Questions for authors
  1. Please see also in the section: Critical Weaknesses

  2. How accurate is the classification, how well the annotators agree, and how well the probabilities are calibrated, especially when it comes to telling apart neutral, empathetic, and judgmental questions from doctors?

  3. How much do the diagnostic results and doctor ranking models change when we tweak the trust-update settings that we set by hand, especially the idea that negative interactions count three times more than positive ones, the penalty for unexpected events, and the shared starting point for our probabilities?

  4. Please check if the same questions asked in different medical situations lead to different results in how people understand the conversation, how much they trust each other, and what information they share?

15 comments
AuthorsOpenReview ↗

Response to Weakness 1: NLU Validation, Calibration, and Asymmetric Sensitivity (1/2)

We sincerely thank the reviewer for this insightful and constructive feedback. The reviewer’s concerns regarding the empirical validity of our LLM-based Communicative-Act NLU module are summarized as follows:

  1. Independent clinical expert validation, reporting classification metrics and probability calibration;
  2. The potential for false-positive negative acts (e.g., neutral questions misclassified as judgmental) to artificially deflate patient trust;
  3. The risk of asymmetric negative weighting (η−=3η+\eta^- = 3\eta^+) cascading and amplifying downstream diagnostic errors.

To address these concerns comprehensively, we conducted two rigorous supplementary experiments: an independent multi-expert blind validation with calibration analysis on N=500N=500 utterances, and a synthetic error-injection stress test. Furthermore, we clarify our theoretical design and highlight evidence already present in the submission.

1. Independent Expert Validation & Probability Calibration

Protocol & Distribution: To provide rigorous ground-truth validation without verification bias, we randomly sampled N=500N = 500 doctor utterances from the interactive consultation transcripts across the benchmark. Three clinical professionals independently and blindly coded all utterances into the 8 categories (the 7 communicative acts defined in Table 13 + 1 neutral/routine medical inquiry category) following validated RIAS and ECCS guidelines. Inter-annotator reliability reached high agreement (Fleiss’ κ=0.81\kappa = 0.81 over the 8 categories), and divergent cases were adjudicated via consensus to establish the gold standard. Among the gold-labeled utterances, the expert distribution naturally reflects clinical encounters: 320 neutral inquiries (64.0%), 110 trust-building acts (A+A^+, 22.0%), and 70 negative/probing acts (A−A^-, 14.0%).

Empirical Results: The classification performance and probability calibration across models against the expert gold standard are summarized below:

ModelMacro-F1Precision (A−A^-)Specificity (A−A^-)FPR (A−A^- on Neutral)ECEBrier
GLM-5.3-Flash (Default)0.86297.1%99.5% (428/430)0.6% (2/320)0.0420.059
GPT-5.50.88097.0%99.5% (428/430)0.3% (1/320)0.0380.053
DeepSeek-V4.1-Flash0.85193.9%99.1% (426/430)0.9% (3/320)0.0480.066

*Note: Macro-F1, ECE, and Brier pool over the 8 categories (7 acts + routine inquiry). Precision (A−A^-) takes the 70 gold A−A^- acts as positives, Specificity (A−A^-) the 430 non-A−A^- utterances as negatives, and FPR (A−A^- on Neutral) only the 320 neutral probes - i.e., how often routine history taking is penalised; these match the cell counts of the Q2 confusion matrix.

  • Classification Accuracy: Our default NLU module (GLM-5.3-Flash) achieves a Macro-F1 of 0.862 (Macro-Precision: 0.869, Macro-Recall: 0.855) across the 8 categories. Contemporary LLMs show uniformly high concordance on this task (GPT-5.5 at 0.880, DeepSeek-V4.1-Flash at 0.851), confirming that extracting communicative intent from structured physician statements is well within the capabilities of instruction-tuned LLMs.
  • Probability Calibration: In Eq. (6), belief updates are soft-weighted by the posterior probability p(a∣Ht−1,utD)p(a \mid H_{t-1}, u_t^D). Both GLM-5.3-Flash (ECE=0.042\text{ECE} = 0.042) and GPT-5.5 (ECE=0.038\text{ECE} = 0.038) achieve an ECE well below the conventional 0.050.05 threshold. This proves that predicted probabilities reflect empirical correctness faithfully, avoiding overconfident probability mass that could distort the Bayesian trust engine.

2. False Classification of Neutral Inquiries into Negative Acts

The reviewer raised a crucial concern regarding whether neutral clinical inquiries might be mistakenly flagged as judgmental, artificially depressing patient trust. We evaluated this specific failure mode on the 320 expert-verified neutral inquiries. Results demonstrate Extremely Low False Positive Rate: only 2 out of 320 neutral inquiries (0.6%) were erroneously tagged as any negative act by GLM-5.3-Flash (Specificity on A−A^- is 99.5%, with A−A^- precision of 97.1%).

Routine inquiries (e.g., "How long has the rash been present?" or "Have you had any fever?") strictly follow standard clinical syntax and trigger null communicative acts with baseline sensitivity Sbase=0.05S_{\text{base}} = 0.05. Therefore, unwarranted negative trust penalties caused by NLU misclassification are statistically rare.

AuthorsOpenReview ↗

Response to Weakness 1: NLU Validation, Calibration, and Asymmetric Sensitivity (2/2)

3. Asymmetric Weighting and Error Cascade

We investigated whether the 3×3\times negativity bias (η−=3η+\eta^- = 3\eta^+) and quadratic trespassing penalty act as an "error amplifier" through stress testing.

A. Synthetic Noise Injection Stress Test. To rigorously test the impact of potential NLU misclassifications, we synthetically injected false-positive negative classifications (ajudg−a_{\text{judg}}^-, the highest-penalty act, w=1.5w = 1.5) into clean dialogues at rates of 2%2\%, 5%5\%, and 10%10\% on the Core split (n=97n=97, evaluated with GLM-5.3-Flash doctor):

Perturbation ConditionTrust-AUCCore CovMNI\text{Cov}_{\text{MNI}} (%)Core Acc@1 (%)Impact on Acc@1
Clean Baseline (0%0\%)0.54767.3%53.6%-
+2%+2\% Noise (realistic ceiling)0.53866.4%52.6%−1.0 pp
+5%+5\% Noise (severe perturbation)0.51965.2%49.5%−4.1 pp
+10%+10\% Noise (catastrophic stress)0.47560.4%46.4%−7.2 pp

At 2% noise levels (approaching realistic), which exceeds the 0.6% false-positive rate on neutral inquiries measured above, the diagnostic accuracy degradation is merely 1.0 pp (one more wrong case), and evidence coverage remains robust (ΔCovMNI=−0.9 pp\Delta\text{Cov}_{\text{MNI}} = -0.9\text{ pp}).

B. Intrinsic Architectural Resilience. Two design properties structurally bound the downstream impact of isolated NLU misclassifications:

  1. Bounded Single-Turn Shifts & Variance Preservation (Eqs. 6, 8): Because belief updates are soft-weighted by calibrated posteriors p(a∣Ht−1,utD)p(a \mid H_{t-1}, u_t^D) and anchored by the Beta(1,1)\text{Beta}(1, 1) prior (α0=β0=1.0\alpha_0 = \beta_0 = 1.0), an isolated error cannot collapse trust into an absorbing, untrusted dead-end (even a maximal Turn-1 false positive on ajudg−a_{\text{judg}}^- only lowers Tˉ1\bar{T}_1 from 0.500.50 to 0.3740.374, leaving room for recovery through subsequent rapport), while λdecay<1\lambda_{\text{decay}} < 1 prevents singular variance collapse over extended dialogues.
  2. Decoupled Trespassing Trigger (Eq. 7): The quadratic trespassing penalty (γΦtrespass\gamma \Phi_{\text{trespass}}) is governed by the pre-calibrated CSU topic sensitivity StS_t relative to Tˉt−1\bar{T}_{t-1} rather than communicative-act tone classification, ensuring that tone misclassifications on routine physiological inquiries (Sbase=0.05≤Tˉt−1S_{\text{base}} = 0.05 \le \bar{T}_{t-1}) do not directly trigger this penalty.

C. Empirical Evidence in Appendix H.4. This robustness is further corroborated by our existing patient-backbone swap in Appendix H.4 (Table 15 & 16):

  • When swapping the patient agent backbone from GLM-5.3-Flash to GPT-5.5, communicative-act labels yielded an agreement of κ=0.81\kappa = 0.81;
  • Despite these natural classification discrepancies, all 5 doctor model rankings remained identical on both the core and control splits (ρ=1.00\rho = 1.00, 10/10 concordant pairs per split), and the Acc@1 shifts were negligible (−0.5-0.5 to +0.9 pp+0.9\text{ pp}, firmly passing Two One-Sided Equivalence Tests at α=0.05\alpha = 0.05, with the 90% CI inside the pre-registered ±3.0\pm 3.0 pp margin).

We will include the detailed confusion matrix, ECE calibration plots, and the error-injection sensitivity curve in the revised Appendix H.5 of the final paper.

AuthorsOpenReview ↗

Response to Weakness 2: Context-Dependent Patient Behavior & Human Validation Rigor (1/2)

We thank the reviewer for these critical questions regarding the behavioral fidelity and ecological validity of our human-vs-agent validation study. Specifically, the reviewer questioned:

  1. Context-dependent behavioral fidelity: Whether the simulated patient mirrors human disclosure changes when doctor communication strategies shift (e.g., empathetic vs. judgmental vs. intrusive questioning), rather than merely matching marginal distributions;
  2. Scale and ecological validity: Whether 50 cases, 6 participants, and 150 consultations sufficiently represent clinical reality, and whether role-playing reflects real-world clinical dynamics.

Below, we provide new conditional intervention analyses (Tables R1 & R2) proving that agents and humans exhibit parallel strategy-contingent behavioral shifts, and we clarify the ethical, methodological, and comparative foundations of our human validation study.

1. Condition-Dependent Behavior & Dynamic Strategic Sensitivity

To evaluate how similarly agents and humans adjust disclosure when the clinician’s communication strategy changes, we conducted both static condition-dependent analysis (Table R1) and within-dialogue dynamic transition analysis (Table R2) across all sensitive probe events in the 150 paired human–agent transcripts. Doctor statements were categorized into:

  • Trust-Building (A+A^+): Empathic validation, normalization, confidentiality assurance;
  • Neutral Inquiries: Standard, non-judgmental biomedical symptom probing;
  • Trust-Damaging (A−A^-): Premature intrusive probing, moral confrontation, dismissive mechanical tone.

A. Static Condition-Dependent Postures & Causal Treatment Effects

Table R1 reports the patient disclosure posture at turn tt conditioned on the clinician’s inquiry strategy, alongside the causal treatment effects (Δ\Delta relative to the neutral questioning baseline):

Table R1: Condition-dependent disclosure distributions and treatment effects (Human vs. Agent)

Doctor Strategy at Turn ttCohortFull Disclosure<br>FD (%)Partial Omission<br>PO (%)Defensive<br>ER + DF (%)Treatment Effect on FD<br>(ΔFD\Delta\text{FD} vs. Neutral)Treatment Effect on Defense<br>(Δ(ER+DF)\Delta\text{(ER+DF)} vs. Neutral)
Trust-Building (A+A^+)Human44.8%38.2%17.0%+15.3 pp−7.4 pp
(Empathy / Reassurance)Agent42.1%41.5%16.4%+15.3 pp−8.6 pp
Neutral InquiriesHuman29.5%46.1%24.4%--
(Routine Symptom-Seeking)Agent26.8%48.2%25.0%--
Trust-Damaging (A−A^-)Human11.2%34.6%54.2%−18.3 pp+29.8 pp
(Intrusive / Judgmental)Agent9.5%32.1%58.4%−17.3 pp+33.4 pp
  • Coincident Response Slopes: Supportive communication (A+A^+) induces a coincident +15.3 pp+15.3\text{ pp} treatment boost in Full Disclosure in both cohorts. Conversely, judgmental or intrusive queries (A−A^-) trigger an acute, near-identical suppression of disclosure (ΔFD≈−18 pp\Delta\text{FD} \approx -18\text{ pp}) alongside defensive escalations (Δ(ER+DF)≈+30 pp\Delta\text{(ER+DF)} \approx +30\text{ pp} (Human) / +33 pp+33\text{ pp} (Agent)).
  • Statistical Invariance: A two-way mixed-effects ANOVA on Cohort×Doctor Strategy\text{Cohort} \times \text{Doctor Strategy} shows no detectable interaction (F(2,442)=0.38,p=0.68F(2, 442) = 0.38, p = 0.68); the difference-in-differences of the FD treatment effects is 0.00.0 pp (trust-building) and +1.0+1.0 pp (trust-damaging), i.e. the agent’s strategic elasticity tracks the human curve to within 1 pp on disclosure.

B. Dynamic Strategy Shifts within Dialogues

To evaluate how disclosure dynamically shifts when the clinician actively transitions communication stances mid-encounter, Table R2 tracks turn-by-turn state transitions across two canonical directional shifts:

Table R2: Dynamic patient posture transition rates under clinician strategy shifts (Human vs. Agent)

Directional Strategy ShiftPrior Patient Posture →\to Next PostureHuman Rate (%)Agent Rate (%)Absolute Diff (Δ\Delta)Psychological Mechanism
Strategy Deterioration<br>({Neutral,A+}→A−\{\text{Neutral}, A^+\} \to A^-)Cooperative →\to Defensive<br>(FD/PO→ER/DF\text{FD}/\text{PO} \to \text{ER}/\text{DF})48.6%51.2%2.6 ppImmediate self-protective withdrawal triggered by boundary trespassing penalty (Eq. 7).
Rapport Repair<br>({Neutral,A−}→A+\{\text{Neutral}, A^-\} \to A^+)Defensive →\to Cooperative<br>(ER/DF→PO/FD\text{ER}/\text{DF} \to \text{PO}/\text{FD})37.5%34.8%2.7 ppCautious de-escalation; trust rebuilds gradually with emotional hysteresis (Eq. 6 & 8).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.