- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces a dataset of patient profiles containing various risk attributes, generated from 753 real online consultation records. To support the use of this dataset, the authors designed three patient simulation strategies: a patient guided by a generic prompt, a risk-hiding patient that discloses risk information only after relevant inquiries, and a risk-replay patient that discloses information according to the risk trajectory of the original consultation. The authors evaluated five large language models under two doctor-interaction settings and found that different risk-disclosure strategies correspond to different safety outcomes, with higher severe-failure rates in the multi-turn settings than in the separately configured single-turn diagnosis control.
Critical strengths
This study investigates interactional safety risks in real-world scenarios that commonly used QA datasets cannot capture. In practice, patients may withhold risk information, a situation that medical QA datasets primarily presenting case information all at once cannot examine.
Critical weaknesses
1. Insufficient reliability and reproducibility of the dataset, patient profile extraction, and core implementation
Although the paper explicitly lists the dataset as one of its contributions, its description of the specific sources, acquisition methods, screening procedures, de-identification process, and final dataset construction for the 753 online consultations is inadequate. Patient profiles are generated using large language models, but the paper does not adequately clarify whether they underwent manual review, nor does it report the accuracy of risk-attribute extraction, manual review criteria, or inter-annotator agreement.
Meanwhile, the core risk-control mechanisms lack sufficient implementation details. The paper does not provide the specific semantic matching rules, prompts, thresholds, or procedures for handling synonymous expressions and questions involving multiple risks in risk-hiding. It also does not fully describe the DialoguePlan construction process, doctor-act classifier, or triggering conditions. At the time of this review, the anonymous repository promised by the authors still contains no usable code or complete prompts. Consequently, readers cannot determine whether the main results arise from reliable risk-control mechanisms or from undisclosed prompts and heuristic rules. This directly affects the verifiability, reproducibility, and credibility of the paper’s conclusions.
2. The safety results provide insufficient support for the proposed mechanisms
The three simulators differ in the amount and timing of information disclosure and in dialogue length. The existing experiments do not distinguish the effects of increased information disclosure from those of using DialoguePlan. Furthermore, although the authors acknowledge that the single-turn and multi-turn protocols are not equivalent, they primarily support their contribution through differences in overall severe-failure rates, without paired failure analyses for the same cases and doctor models. The evidence is therefore insufficient to establish that multi-turn evaluation reveals interaction-level safety failures missed by single-turn evaluation.
Questions for authors
-
Table 9 presents the taxonomy used to extract risk attributes from source consultations into patient profiles, but Table 7 reports only the frequencies of specific attributes, while Table 6 uses a different set of consultation-level risk categories. Could the authors clarify how the extracted risk attributes map to the categories in Table 9 and how the classification schemes in Tables 6 and 9 relate to each other?
-
How is the semantic matching policy in risk-hiding implemented? What criteria and thresholds determine whether a doctor’s question matches a particular elicitation intent? How does the policy handle synonymous expressions, indirect inquiries, questions involving multiple risks, and cases where the same risk already appears in the medical history?
-
What are the implementation details of DialoguePlan?
-
At the time of this review, the anonymous repository still contains no usable code, complete prompts, or configuration files. Could the authors provide the complete implementation and example intermediate outputs for generating profiles, running risk-hiding, constructing DialoguePlan, classifying doctor acts, and calculating disclosure rates?