- Rating
- 2(−1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how consultation performance changes when the underlying clinical facts remain fixed but patient communication becomes more challenging. It introduces InteractShift, which pairs 147 clinical cases with one Cooperative and five Difficult interaction profiles. The evaluation combines six consultation-level metrics with a fact-level audit separating doctor inquiry from patient disclosure. Across nine doctor models and 7,938 consultations, difficult interactions reduce information acquisition more than history-domain coverage and expose profile-specific and model-specific weaknesses. Two expert studies assess simulation quality, reproduce the main degradation pattern, and validate automatic capability and fact-level judgments.
Critical strengths
-
The problem setting controls clinical content while exposing an important source of evaluation variation.
-
The evaluation is supported by substantial and complementary expert assessment.
Critical weaknesses
-
Doctor limitations and simulator-imposed difficulty remain entangled. Profiles explicitly prescribe restricted disclosure, but no intervention establishes how much of the loss better consultation could avoid. One stochastic patient simulator limits generalizability; expert-rated profile consistency does not establish realistic responsiveness to effective questioning.
-
Some metric comparisons require qualification. COVERAGE and ELICITATION use different units and granularity, so their point drops are not calibrated measures of relative capability loss. RESILIENCE assigns full credit when no misleading incident occurs, conflating successful resistance with lack of exposure.
-
The fact audit remains descriptive rather than mechanistic. ASKED includes inquiries after disclosure, and inquiry-status groups contain different facts across profiles. Consequently, conditional disclosure rates do not establish whether specific questioning improves information acquisition; the proposed repair strategies remain untested.
-
Lack of novelty. The entire work only identifies problems and designs analyses to systematically evaluate multiple LLMs, and those findings are been exposed in previous works, but it does not propose a new solution. Have you considered the following situation: when a patient is very defensive, they may lie? For example, when asked whether they are pregnant, they may lie.
Questions for authors
-
Could a representative subset be repeated with a second patient simulator and multiple generation seeds? Do profile-specific effects and model rankings persist?
-
At matched dialogue states with an undisclosed fact, could the authors compare broad inquiry, specific inquiry, and specific inquiry with profile-appropriate communication support? This would test whether effective adaptation can overcome the simulator’s disclosure restrictions.
-
Could the fact audit distinguish inquiries preceding first disclosure from subsequent confirmation and compare matched facts across profiles? For RESILIENCE, please report incident exposure separately from performance conditional on exposure.
-
Does the benchmark cover intentional false reporting, rather than withholding or vagueness? Since the simulator is instructed not to alter facts, please distinguish controlled deception from simulator factual errors and clarify the scope of the resilience findings.
-
Could Figure 1 include a turn-aligned worked example showing how paired consultations receive different inquiry and disclosure labels? The distinction between complete-information ELICITATION and partial-but-useful CORRECT disclosure should also be clearer in the main text.