acceptodds
Desk rejected at ICLR 2027

ElicitDoctor: Robust Multi-Turn Evidence Elicitation in Psychiatric Dialogue under Low-Disclosure Patient Responses

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Psychiatric dialogue requires evidence elicitation: asking about a relevant topic does not ensure that a patient's reply supports an assessment. We introduce ElicitBench, a controlled benchmark for multi-turn evidence elicitation under low-disclosure responses, constructed from MDD-5K and DAIC-family corpora. It separates dialogue-derived hidden profiles, stateful disclosure control, doctor-visible interaction, and evaluator-side evidence recovery. Its evaluation protocol combines construction-quality checks, full-trajectory patient audits, scoring validation, and matched doctor-policy comparisons. We also propose ElicitDoctor, a question-generation policy trained with immediate visible-dialogue feedback and continuation value through group-relative optimization. The experimental design examines evidence coverage, acquisition efficiency, boundary-respecting behavior, and the relationship between recovered evidence and downstream diagnostic or screening judgment. Together, the benchmark and method provide a framework for studying adaptive inquiry without exposing hidden case evidence to the doctor.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
interactive diagnosis, evidence elicitation, patient role-play

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_unETOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ElicitDoctor studies multi-turn psychiatric evidence elicitation under low-disclosure patient responses, where relevant questions may receive vague, partial, or refusing answers. The authors construct a disclosure-controlled patient simulator using MDD-5K and DAIC-family data and propose a belief-guided GRPO method that combines immediate question-resolution feedback with a learned estimate of longer-term inquiry value. Experiments report improved evidence coverage over supervised fine-tuning under restricted disclosure and separately examine patient-simulation quality and the diagnostic usefulness of elicited conversations.

Critical strengths
  • This paper addresses a clear and practically meaningful problem. By distinguishing topic coverage from evidence acquisition, the paper proposes a more appropriate consultation objective for vague, partial, and refusing responses.
  • Provides a coherent approach to interaction modeling. The method distinguishes unknown information from explicit denial and combines immediate question resolution with longer-term inquiry value to adapt subsequent questions.
Critical weaknesses
  • Patient realism and the clinical quality of the trained doctor remain insufficiently validated. Disclosure dynamics rely on manually specified rules rather than empirically calibrated patient behavior, and patient-quality evaluation relies on LLM judges. The paper reports no systematic independent clinical expert assessment of either simulated patients or the trained doctor’s questioning behavior. Consequently, higher evidence coverage may reflect adaptation to simulator-specific rules rather than clinically useful inquiry. Experimental suggestions: conduct blinded psychiatric-expert evaluations of patient realism, evidence fidelity, and doctor behavior (including inappropriate persistence after refusal), and test whether gains transfer to independently configured patient simulators.
  • The algorithmic contribution is insufficiently distinguished from established approaches. ElicitDoctor combines LLM-based state assessment, reward shaping, rollout-based value learning, and GRPO, while the Long signal is derived from cumulative Short rewards. The paper does not sufficiently establish what this formulation contributes beyond standard reward shaping and long-horizon value guidance.
Questions for authors
  • Do the 142 overlapping participants introduce shared patient facts or evaluation targets across training/development and testing? Please report the performance of ElicitDoctor over baselines separately for seen and unseen (OOD) participants, ideally with a participant-disjoint evaluation.
  • Appendix E.3 states that ElicitDoctor and its local baselines are excluded from the uniform diagnostic reassessment. Could the authors evaluate the final ElicitDoctor checkpoint and these baselines using the same evaluator, prompt, and matched test cases, to determine whether improved evidence coverage translates into better diagnostic performance?
2 comments

The stratified results are shown below. Each cell reports ECR / Macro-F1 (%). Within each subgroup and disclosure condition, all three policies are evaluated on the same participants with valid completed interviews and diagnostic predictions. N denotes this matched sample size.

Participant groupConditionQwen3-8B BaseSFTElicitDoctor (Full)
Non-overlapping (77 IDs)Zero46.83 / 68.2090.30 / 75.1992.47 / 80.86
Non-overlapping (77 IDs)Mild43.84 / 65.3964.55 / 74.2666.25 / 77.84
Non-overlapping (77 IDs)Moderate50.58 / 68.8751.35 / 73.2157.00 / 73.77

Full has numerically higher Macro-F1 than Base in both participant groups under all three conditions. Relative to SFT, Full has higher Macro-F1 across all conditions in the overlapping group. In the non-overlapping group, it performs better under Zero.

Uniform diagnostic reassessment of ElicitDoctor and local baselines

Thank you for this suggestion. We have extended the independent diagnostic evaluation to the available Full checkpoint and the Qwen3-8B Base/SFT controls, alongside the four closed-source doctors reported in the paper. All diagnoses use the official Flash evaluator and the same diagnostic system prompt, based only on the visible interview.

The evaluation covers the full 219-participant DAIC inventory. Each cell reports ECR / Macro-F1 (%), calculated on the same valid dialogues within each policy–condition group. Valid-case counts differ across policies because unsuccessful interviews are excluded.

Doctor policyZero / NaturalMildModerate
DeepSeek V4 Pro83.16 / 79.3670.06 / 75.5155.34 / 66.33
GPT-5.6-terra70.21 / 75.5650.11 / 65.5741.78 / 65.03
Qwen3.8-Flash74.60 / 76.8549.49 / 64.6034.25 / 65.34
GPT-5-nano33.50 / 66.9228.31 / 57.9823.29 / 51.14
Qwen3-8B Base47.59 / 67.3248.62 / 63.8441.98 / 66.18
SFT90.02 / 73.7865.58 / 70.7750.49 / 69.10
ElicitDoctor (Full)91.44 / 77.5865.79 / 69.1257.34 / 71.38

I appreciate the authors' thorough rebuttal and the extra validations provided during the discussion period. However, taking into account the overall contribution, I feel my initial rating already represents the upper bound for this submission. Therefore, I will maintain my score.

Reviewer_xJ6wOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies psychiatric dialogue in which a relevant question may receive a vague, partial, or refusing answer. It builds a disclosure-controlled simulator from MDD-5K and DAIC-family sources, separating hidden case information from doctor-visible dialogue. ElicitDoctor trains a Qwen3-8B doctor with GRPO using a frozen assessor's immediate question-resolution reward and a learned ranking score for cumulative inquiry returns under a fixed continuation policy. Evaluation considers source-grounded patient role-play, source-target evidence coverage (ECR), and separate diagnostic judgment. The reported local-policy summaries suggest improved low-disclosure recovery, while the Full results are interim. The uniform diagnostic reassessment covers four closed-source doctors, rather than the final ElicitDoctor policy.

Critical strengths
  1. A valuable problem formulation with explicit information boundaries. The paper addresses an important limitation of interactive diagnosis: asking a clinically relevant question does not mean that the requested evidence has been obtained. Its distinction among unknown information, explicit negatives, partial answers, and adequate answers provides a useful basis for studying adaptive follow-up. Separating the hidden patient profile and evaluation targets from the doctor-visible dialogue and belief tracker further clarifies the learning problem (Sections 2.1–2.2; Appendix B.1).

  2. An integrated interaction architecture and useful analytical results. The framework connects patient simulation, question-resolution assessment, continuation-value learning, policy optimization, and evaluation in a concrete research pipeline (Sections 2–3). The descriptions of the controller, prompts, and assessment rubrics provide an actionable methodological starting point for subsequent work. The response-form analysis, original-source quality comparison, and separation of evidence recovery from diagnostic judgment are informative. In particular, reporting naturalness limitations and differing recovery–diagnosis relationships helps identify where the simulator and inquiry objective need further refinement.

Critical weaknesses
  1. Patient realism and the interpretation of learned strategies lack sufficient independent validation. The source-grounding constraints, two-LLM trajectory review, and original-source comparisons are useful controls (Table 1; Appendix B.2). However, source consistency and model-assigned quality do not establish whether the simulated changes in disclosure, refusal, and readiness are behaviorally plausible. The reported validation contains no human assessment of the actual generated doctor–patient trajectories, while the same controller assumptions shape both training interactions and evaluation. Consequently, it remains unclear whether effective strategies reflect broadly plausible low-disclosure inquiry or depend strongly on this particular patient environment. This limits confidence in the benchmark and policy-efficacy interpretation, even within the stated simulation scope. A bounded human review of complete generated trajectories would address this concern without requiring a clinical trial.

  2. The results are insufficiently characterized across heuristic controller and reward choices. Trust/readiness thresholds and state updates are manually specified, while the short-term utility and interaction costs use fixed coefficients, including the 0.5 partial-answer weight and refusal/termination penalties (Appendices B.1 and C.2). These choices have understandable semantic motivations, but their numerical adequacy and influence on the conclusions are not established by controlled sensitivity results. The paper therefore leaves unresolved how stable the training benefit is under other reasonable configurations. A limited development-set search over short-term reward settings, together with separate perturbations of controller parameters and fixed held-out evaluation, would be informative. Configuration quality should include independently assessed interaction quality and respect for boundaries, rather than selecting only the largest ECR.

Questions for authors

Could you provide a focused, human-validated robustness analysis showing whether ElicitDoctor's gains persist relative to the existing same-size Qwen3-8B Base and SFT controls, using the same cases, disclosure conditions, and 24-turn budget?

(a) Sample complete doctor–patient trajectories actually generated by these policies and have multiple human raters, preferably with relevant clinical expertise, assess patient source fidelity, the plausibility of low disclosure/refusal/readiness changes, and—separately—doctor inquiry quality and boundary respect. Blind policy identity and condition labels where feasible, and use clinical plausibility rather than controller compliance alone as the reference.

(b) Perform a bounded development-set search over short-term reward coefficients and separately test sensitivity to disclosure-controller thresholds/state updates; freeze the selected settings before held-out evaluation. Report whether the trained-policy advantage remains stable across reasonable settings, including interaction-quality trade-offs.

(c) Have humans and the LLM judges score the same sampled trajectories with aligned rubrics, report human–human and human–LLM agreement by dimension, and show whether the policy ranking is supported by the human assessments. These results could materially change my assessment of both simulator validity and the claimed training benefit.

1 comment

Thank you for highlighting the importance of independent validation. Patient simulation is an established research approach for interactive medical AI: AgentClinic studies information gathering and diagnosis under incomplete information [1], while PATIENT-Ψ supports mental-health training through simulated patient interactions [2].

Our question is specific: when case facts are fixed and disclosure is controlled, can a doctor acquire evidence through adaptive follow-up? This setting makes questioning strategies measurable beyond whether they merely mention relevant topics. We do not claim to reproduce the full behavioral distribution of real psychiatric patients.

(a) Source fidelity and human validation

Our evidence targets are traceable: MDD targets link to patient statements and their dialogue context, while DAIC targets derive from PHQ-8 items. Mentioning a topic alone earns no recovery credit. The paper provides source-record comparisons and two-LLM trajectory assessments; ongoing human annotation checks source support and evidence representation.

This annotation concerns evidence construction, not the clinical realism of complete interactions. Our planned trajectory review would compare Full, Base, and SFT on matched cases, disclosure conditions, and a 24-turn maximum budget.

(b) Why fixed reward and controller settings

The reward coefficients express explicit task preferences, while controller thresholds define reproducible disclosure conditions rather than empirically calibrated psychological parameters. Our experiments focus on policy performance within this specified task. We therefore did not conduct a systematic parameter search in this round.

Such a search is not a prerequisite for defining and evaluating the controlled task, but it is needed to support broader claims of robustness across parameter choices. Our existing ablations address reward-component contributions, not coefficient sensitivity. We accordingly keep our conclusions conditional on the reported settings. Bounded development-set reward searches and separate controller perturbations with frozen doctor checkpoints are follow-up tests of this broader robustness.

(c) Human–LLM agreement

The planned human review would use the same sampled trajectories, source information, and dimension-specific rubrics as the LLM assessment. We would report human–human and human–LLM agreement from independent ratings and examine whether human judgments support the policy rankings.

These trajectory-level human assessments and agreement analyses have not yet been completed. Our current contribution remains a source-grounded, disclosure-controlled framework with task-level evaluation; we do not present it as already human-validated clinical interaction.

References

[1] Schmidgall, S., et al. (2024). AgentClinic: A multimodal agent benchmark to evaluate AI in simulated clinical environments. Link

[2] Wang, R., et al. (2024). PATIENT-Ψ: Using Large Language Models to Simulate Patients for Training Mental Health Professionals. EMNLP 2024. Link

Reviewer_oxQ2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multi-turn evidence elicitation in psychiatric dialogue. To improve adaptive questioning under low-disclosure conditions, the authors introduce ElicitDoctor, a belief-guided reinforcement learning framework. It evaluates evidence acquisition and downstream diagnostic performance, showing that ElicitDoctor improves low-disclosure ECR over supervised fine-tuning (SFT). Ablation results also suggest benefits from combining short-term and long-term feedback.

Critical strengths
  • The paper addresses an important limitation of existing interactive diagnostic systems which ask an appropriate question does not necessarily produce useful information.
  • The proposed reward design is reasonable, it captures both the immediate information gained the potential benefits of a question for subsequent interactions.
Critical weaknesses
  • The experiments rely on an LLM-based patient simulator. All patient simulator assessments are conducted by LLM judges, without human or clinical expert verification. For example, validity rests on two LLM judges with low exact agreement, and simulated MDD naturalness is markedly below raw records
  • The Full ElicitDoctor results and the DAIC Long-only results in Table 2 are marked as interim . Without complete paired comparisons, it is difficult to determine how reliably the combination rewards outperforms its individual components.
  • Whether ElicitDoctor yields more diagnostically useful dialogues is untested. ElicitDoctor is excluded from the uniform evaluation (RQ3).
  • The authors claim belief-guided training improves low-disclosure recovery. But closed-source models still outperform ElicitDoctor on MDD low-disclosure mean (16.08–29.20 compared to 14.59).
Questions for authors
  • For the patient simulator, can the authors provide human or clinical expert evaluations to verify its realism and reliability?
  • Can the authors provide the final results for Full ElicitDoctor and DAIC Long-only, with statistical significance test?
  • Can the authors evaluate whether the improved evidence recovery of ElicitDoctor also leads to better diagnostic performance?
  • How do the authors explain the gap on MDD under low disclosure, and what supports the effectiveness of the proposed method beyond improvements over SFT baseline?
2 comments

1. Patient simulation, human validation, and future evaluation

Thank you for highlighting the importance of independent validation. Patient simulation has been used to study interactive medical AI: AgentClinic evaluates information gathering and diagnosis under incomplete information [1], while PATIENT-Ψ uses patient cognitive models to support simulated interactions for mental health training [2].

Our work focuses on a specific question: can a doctor acquire evidence through adaptive follow-up when case facts are fixed and disclosure is controlled? This environment enables controlled comparisons of questioning policies. We do not claim that it reproduces the full behavioral distribution of real psychiatric patients.

Our evidence targets have traceable sources. MDD targets derive from patient statements in the original dialogues, interpreted using the preceding questions and local context, with source locations retained. DAIC targets are constructed from PHQ-8 items. Merely asking about a topic does not count as recovering its evidence. We are conducting human annotation to verify source support and representational accuracy. This annotation addresses evidence construction; it does not establish the clinical realism of simulated disclosure, refusal, or complete interactions.

Related work also informs our evaluation plans. PATIENT-Ψ includes a user study with mental health experts and trainees [2]. AMIE evaluates diagnostic dialogue through a randomized, double-blind crossover study with validated patient-actors, assessing history-taking, diagnosis, communication, and empathy separately [3]. These designs motivate evaluating task performance and interaction quality as distinct dimensions.

2. Updated evidence recovery results The results are summarized below.

DatasetPolicyZero / NaturalMildModerateMean
DAICFull91.4465.7957.3461.56
DAICShort-only92.6963.7550.0056.88
DAICLong-only90.9267.2948.6957.99
MDD-5KFull50.3927.5320.1723.85
MDD-5KShort-only42.2212.297.579.93
MDD-5KLong-only45.4426.5518.6422.60
3. Does improved evidence recovery translate into better diagnostic performance?

Thank you for this suggestion. We have extended the independent diagnostic evaluation to the available Full checkpoint and local Base/SFT controls, alongside the four closed-source doctors reported in the paper. All diagnoses use the official Flash evaluator and the same diagnostic system prompt, based only on the visible interview.

The following results cover the full 219-participant DAIC inventory. Each cell reports ECR / Macro-F1 (%).

Doctor policyZero / NaturalMildModerate
DeepSeek V4 Pro83.16 / 79.3670.06 / 75.5155.34 / 66.33
GPT-5.6-terra70.21 / 75.5650.11 / 65.5741.78 / 65.03
Qwen3.8-Flash74.60 / 76.8549.49 / 64.6034.25 / 65.34
GPT-5-nano33.50 / 66.9228.31 / 57.9823.29 / 51.14
Qwen3-8B Base47.59 / 67.3248.62 / 63.8441.98 / 66.18
SFT90.02 / 73.7865.58 / 70.7750.49 / 69.10
ElicitDoctor (Full)91.44 / 77.5865.79 / 69.1257.34 / 71.38

Overall,Higher evidence recovery can accompany better diagnosis

4. Competitive performance with a locally deployable 8B doctor

Thank you for this question. Beyond improvement over SFT, our supplementary MDD results demonstrate the competitiveness of a Qwen3-8B-based doctor against several closed-source baselines.

An additional practical advantage is local deployment in privacy-sensitive settings. The trained 8B doctor can generate follow-up questions locally, allowing doctor inference to remain within an institution-controlled environment and reducing reliance on external doctor APIs for processing sensitive psychiatric dialogue. This deployment option is valuable alongside task performance: the objective is to obtain an effective, locally deployable questioning policy. Together, these findings support the practical value of task-specific training for a locally deployable 8B doctor. References

[1] Schmidgall, S., Ziaei, R., Harris, C., Reis, E., Jopling, J., and Moor, M. (2024). AgentClinic: A multimodal agent benchmark to evaluate AI in simulated clinical environments. arXiv:2405.07960. Link

[2] Wang, R., et al. (2024). PATIENT-Ψ: Using Large Language Models to Simulate Patients for Training Mental Health Professionals. Proceedings of EMNLP 2024, pp. 12772–12797. Link

[3] Tu, T., Schaekermann, M., Palepu, A., et al. (2025). Towards conversational diagnostic artificial intelligence. Nature, 642, 442–450. Link

Thank you for the rebuttal. Overall, the rebuttal has improved my understanding of the method and the experimental evidence. However, the remaining concerns regarding simulator validation and statistical reliability are critical to the paper's claims, and I understand that they cannot be fully addressed within the discussion period. I believe the paper would be substantially strengthened by a major revision that carefully addresses these concerns. Therefore, I will maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.