acceptodds
Under review as a conference paper at ICLR 2027

InteractShift: Diagnosing Doctor LLM Consultation Degradation under Difficult Interactions via Fact-Anchored Paired Stress Testing

Abstract

Performance with cooperative patients does not establish how doctor LLM consultation capabilities change when interactions become difficult. We introduce InteractShift, a fact-anchored paired stress test that holds clinical facts fixed while varying interaction profiles. Across 147 clinical cases, one Cooperative and five Difficult interaction profiles, and nine doctor LLMs, we evaluate six consultation capabilities and separately audit doctor-side elicitation and patient-side disclosure at the fact level. Difficult interactions produce significant but uneven degradation: history-taking Coverage decreases by 6.3 points, whereas clinical information Elicitation decreases by 24.9 points on a 0-100 scale. Fact auditing shows more target facts receive explicit or partial inquiry under Difficult conditions despite lower domain-level coverage, yet correct disclosure is lower within each doctor-side elicitation status. Explicit inquiry is associated with smaller Cooperative-Difficult gaps in correct disclosure, although substantial deficits remain under Vague and Overwhelmed interactions. Different interaction profiles stress different consultation capabilities, while models differ in retained capabilities and shifts in doctor-side elicitation. Complementary expert studies reproduce the main degradation pattern, support patient simulation quality, and show agreement with automatic evaluation at both capability and fact levels. InteractShift provides a controlled basis for diagnosing where consultation capabilities degrade and information acquisition breaks down.

Primary area:
datasets and benchmarks
Keywords:
Medical LLMs, Clinical Dialogue, Interactive Evaluation, Patient Simulation, LLM Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pftAOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how consultation performance changes when the underlying clinical facts remain fixed but patient communication becomes more challenging. It introduces InteractShift, which pairs 147 clinical cases with one Cooperative and five Difficult interaction profiles. The evaluation combines six consultation-level metrics with a fact-level audit separating doctor inquiry from patient disclosure. Across nine doctor models and 7,938 consultations, difficult interactions reduce information acquisition more than history-domain coverage and expose profile-specific and model-specific weaknesses. Two expert studies assess simulation quality, reproduce the main degradation pattern, and validate automatic capability and fact-level judgments.

Critical strengths
  1. The problem setting controls clinical content while exposing an important source of evaluation variation.

  2. The evaluation is supported by substantial and complementary expert assessment.

Critical weaknesses
  1. Doctor limitations and simulator-imposed difficulty remain entangled. Profiles explicitly prescribe restricted disclosure, but no intervention establishes how much of the loss better consultation could avoid. One stochastic patient simulator limits generalizability; expert-rated profile consistency does not establish realistic responsiveness to effective questioning.

  2. Some metric comparisons require qualification. COVERAGE and ELICITATION use different units and granularity, so their point drops are not calibrated measures of relative capability loss. RESILIENCE assigns full credit when no misleading incident occurs, conflating successful resistance with lack of exposure.

  3. The fact audit remains descriptive rather than mechanistic. ASKED includes inquiries after disclosure, and inquiry-status groups contain different facts across profiles. Consequently, conditional disclosure rates do not establish whether specific questioning improves information acquisition; the proposed repair strategies remain untested.

  4. Lack of novelty. The entire work only identifies problems and designs analyses to systematically evaluate multiple LLMs, and those findings are been exposed in previous works, but it does not propose a new solution. Have you considered the following situation: when a patient is very defensive, they may lie? For example, when asked whether they are pregnant, they may lie.

Questions for authors
  1. Could a representative subset be repeated with a second patient simulator and multiple generation seeds? Do profile-specific effects and model rankings persist?

  2. At matched dialogue states with an undisclosed fact, could the authors compare broad inquiry, specific inquiry, and specific inquiry with profile-appropriate communication support? This would test whether effective adaptation can overcome the simulator’s disclosure restrictions.

  3. Could the fact audit distinguish inquiries preceding first disclosure from subsequent confirmation and compare matched facts across profiles? For RESILIENCE, please report incident exposure separately from performance conditional on exposure.

  4. Does the benchmark cover intentional false reporting, rather than withholding or vagueness? Since the simulator is instructed not to alter facts, please distinguish controlled deception from simulator factual errors and clarify the scope of the resilience findings.

  5. Could Figure 1 include a turn-aligned worked example showing how paired consultations receive different inquiry and disclosure labels? The distinction between complete-information ELICITATION and partial-but-useful CORRECT disclosure should also be clearer in the main text.

Reviewer_P1NfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper evaluates nine LLMs on a patient simulator built on Qwen3-32B with five difficult interaction profiles. It evaluates six consultation capabilities and provide clinician annotations.

Critical strengths
  1. The paper addresses an important medical problem.
  2. The human validation can be useful for future LLM evaluation or LLM-as-judge studies.
Critical weaknesses
  1. My main concern is that the novelty appears limited. Many existing works already evaluate LLMs using simulated patients with different personas. For example, PatientSim includes distrustful, impatient, and overanxious personas, along with variations in language proficiency, medical history recall, and cognitive confusion. Several proposed personas appear similar despite differences in names or definitions, and it is unclear what substantially different findings they yield. MAQuE already incorporates Atomic Information Units, Disclosure Control, Linguistic Variation, and Noise Injection, and reports "Controlling the patient’s disclosure behavior caused a marked decrease in information coverage". MedDialBench and VeriSim also study performance changes under difficult patient communication. The paper needs to clarify which important findings go beyond what these existing frameworks provide.

  2. The findings concern simulated patients rather than real patients, and all experiments use a single patient simulator, Qwen3-32B. There is insufficient evaluation of how closely the simulator resembles real patients. Clinician annotations assess factual fidelity and persona consistency, but do not directly establish whether the responses resemble those of real patients. It also remains unclear whether the findings are specific to Qwen3-32B or hold across other patient LLMs.

  3. The main finding is largely expected, and it is difficult to separate the effects of doctor capability from simulator behavior. Difficult patients are prompted to provide vague, delayed, or incomplete information, so reduced information acquisition is unsurprising. The analysis is mainly descriptive: it does not test strategies for overcoming these barriers or examine how information acquisition relates to diagnosis or safety.

  4. Coverage assigns equal weight to all six history-taking domains and requires all six for a full score, without considering their relevance or necessity for each case. Not every case necessarily requires inquiry into all six domains, so this metric may not directly reflect clinical consultation quality.

Questions for authors
  1. What substantive insights does InteractShift provide beyond PatientSim, MAQuE, MedDialBench, and VeriSim?
  2. What evidence supports the simulator’s realism and the findings’ robustness beyond Qwen3-32B?
  3. How do you distinguish doctor-side limitations from disclosure restrictions imposed by the simulator?
  4. Why does full Coverage require all six domains with equal weights, regardless of their relevance to each case?
Reviewer_2WB6OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary
  • The authors introduce InteractShift to evaluate how consultations conducted by doctor LLMs change when the underlying clinical facts are held fixed and patient interaction profiles vary. They evaluate nine doctor LLMs on 147 clinical cases, with one cooperative and five difficult profiles for each case.

  • The evaluation uses six capability metrics and a separate fact-level audit of whether doctors ask about each fact and whether patients disclose it. Overall, difficult interactions reduce scores on all six metrics. Information acquisition declines even though a larger share of target facts receives direct or partial inquiry.

  • Two expert studies assess the results and automatic evaluation. Physicians assess whether the degradation pattern persists under a separate scoring protocol and rate factual fidelity and profile consistency. A second study compares the automatic capability and fact-level labels with adjudicated expert annotations.

Critical strengths
  1. By holding clinical facts fixed across interaction conditions, InteractShift enables within-case comparisons of what doctor LLMs ask about and what information they actually obtain (Sections 3.4 and 4.2).

  2. Physicians observed the same broad pattern of performance degradation using a separate scoring protocol. The automated inquiry and disclosure labels also showed high agreement with adjudicated expert annotations (Section 5 and Appendix D).

Critical weaknesses

I have three main concerns about the contribution, evaluation design, and interpretation of the results.

  1. (contribution) The paper does not clearly establish what its analysis adds to prior work. Previous studies have already documented performance degradation under difficult patient interactions and examined failures in questioning and elicitation. InteractShift separately analyzes whether doctors ask about each fact and whether patients disclose it. However, the paper does not clearly explain what additional findings this analysis provides beyond previous studies.

  2. (evaluation design) The main text does not clearly explain how the six capability metrics were designed or how they are used to evaluate consultations. Readers need to consult the appendix to understand the scoring units, aggregation rules, and relationship between these metrics and the separate fact-level audit. These details are necessary to interpret the reported results and should be explained more clearly in the main text.

  3. (interpretation of the results) It is unclear how much of the decline in information acquisition reflects limitations in doctor behavior and how much follows from the simulator's disclosure rules. The expert checks assess factual fidelity and consistency with the assigned profiles, but they do not establish whether clinically appropriate follow-up can elicit the target facts. Without this validation, it is difficult to determine how much of the information loss better consultation could prevent. The fact-level audit also counts questions asked after the relevant information has already been disclosed. A higher inquiry rate therefore does not by itself show that doctors adapted effectively to difficult interactions (Section 4.2 and Appendices A and C.3).

Questions for authors
  1. (about contribution) What conclusions does InteractShift's evaluation support beyond prior analyses of information acquisition and consultation failures?

  2. (about clinical grounding) How did the clinical literature inform the five profiles and their disclosure rules? Can target facts remain undisclosed despite appropriate follow-up, and how was the simulator's response to such follow-up validated?

  3. (about additonal analysis) Does the higher share of queried facts persist at comparable turn budgets? How often does follow-up after an inadequate answer lead to new disclosure? How do dialogue lengths, maximum-turn reach rates, and termination patterns vary across profiles and doctor models?

  4. (about rubric design and human evaluation) What roles did clinicians and LLMs play in developing and revising the rubrics? Why did Study 1 use different definitions and scoring units, and what can it establish about the automated metrics beyond reproducing the direction of degradation? How were the 810 dialogues allocated across the ten physicians, and how many physicians rated each dialogue?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.