- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ElicitDoctor studies multi-turn psychiatric evidence elicitation under low-disclosure patient responses, where relevant questions may receive vague, partial, or refusing answers. The authors construct a disclosure-controlled patient simulator using MDD-5K and DAIC-family data and propose a belief-guided GRPO method that combines immediate question-resolution feedback with a learned estimate of longer-term inquiry value. Experiments report improved evidence coverage over supervised fine-tuning under restricted disclosure and separately examine patient-simulation quality and the diagnostic usefulness of elicited conversations.
Critical strengths
- This paper addresses a clear and practically meaningful problem. By distinguishing topic coverage from evidence acquisition, the paper proposes a more appropriate consultation objective for vague, partial, and refusing responses.
- Provides a coherent approach to interaction modeling. The method distinguishes unknown information from explicit denial and combines immediate question resolution with longer-term inquiry value to adapt subsequent questions.
Critical weaknesses
- Patient realism and the clinical quality of the trained doctor remain insufficiently validated. Disclosure dynamics rely on manually specified rules rather than empirically calibrated patient behavior, and patient-quality evaluation relies on LLM judges. The paper reports no systematic independent clinical expert assessment of either simulated patients or the trained doctor’s questioning behavior. Consequently, higher evidence coverage may reflect adaptation to simulator-specific rules rather than clinically useful inquiry. Experimental suggestions: conduct blinded psychiatric-expert evaluations of patient realism, evidence fidelity, and doctor behavior (including inappropriate persistence after refusal), and test whether gains transfer to independently configured patient simulators.
- The algorithmic contribution is insufficiently distinguished from established approaches. ElicitDoctor combines LLM-based state assessment, reward shaping, rollout-based value learning, and GRPO, while the Long signal is derived from cumulative Short rewards. The paper does not sufficiently establish what this formulation contributes beyond standard reward shaping and long-horizon value guidance.
Questions for authors
- Do the 142 overlapping participants introduce shared patient facts or evaluation targets across training/development and testing? Please report the performance of ElicitDoctor over baselines separately for seen and unseen (OOD) participants, ideally with a participant-disjoint evaluation.
- Appendix E.3 states that ElicitDoctor and its local baselines are excluded from the uniform diagnostic reassessment. Could the authors evaluate the final ElicitDoctor checkpoint and these baselines using the same evaluator, prompt, and matched test cases, to determine whether improved evidence coverage translates into better diagnostic performance?
2 comments
I appreciate the authors' thorough rebuttal and the extra validations provided during the discussion period. However, taking into account the overall contribution, I feel my initial rating already represents the upper bound for this submission. Therefore, I will maintain my score.
The stratified results are shown below. Each cell reports ECR / Macro-F1 (%). Within each subgroup and disclosure condition, all three policies are evaluated on the same participants with valid completed interviews and diagnostic predictions. N denotes this matched sample size.
Full has numerically higher Macro-F1 than Base in both participant groups under all three conditions. Relative to SFT, Full has higher Macro-F1 across all conditions in the overlapping group. In the non-overlapping group, it performs better under Zero.
Uniform diagnostic reassessment of ElicitDoctor and local baselines
Thank you for this suggestion. We have extended the independent diagnostic evaluation to the available Full checkpoint and the Qwen3-8B Base/SFT controls, alongside the four closed-source doctors reported in the paper. All diagnoses use the official Flash evaluator and the same diagnostic system prompt, based only on the visible interview.
The evaluation covers the full 219-participant DAIC inventory. Each cell reports ECR / Macro-F1 (%), calculated on the same valid dialogues within each policy–condition group. Valid-case counts differ across policies because unsuccessful interviews are excluded.