acceptodds
Under review as a conference paper at ICLR 2027

PCQC: Privileged Counterfactual Question Credit for Multi-Turn Medical Dialogue

Abstract

Large language models (LLMs) have made substantial progress on medical question-answering, yet effective medical dialogue also requires learning to ask questions that uncover relevant patient information. To train such dialogue policies, a common pipeline combines supervised fine-tuning with reinforcement learning (RL) based on final diagnostic correctness. However, this outcome-based supervision does not directly distinguish the contributions of individual questions and provides no question-level feedback for unexecuted alternatives. To address this gap, we introduce PCQC (Privileged Counterfactual Question Credit), which uses privileged patient information during training to learn from questions never asked. During training, PCQC makes alternative questions directly comparable at the same dialogue state by using privileged patient facts to construct their answers. A frozen diagnostic scorer evaluates the diagnostic utility of each resulting question–answer pair by how strongly it supports the correct diagnosis. PCQC turns these comparisons into relative question credit that teaches the policy which questions to favor, directly supervising both executed and unexecuted questions alongside outcome-based RL without requiring complete rollouts for the unexecuted alternatives. Extensive experiments across four medical benchmarks demonstrate that PCQC achieves 63.10% mean diagnostic accuracy, outperforming GRPO and ATPO by 4.38 and 4.21 percentage points, respectively. These gains are achieved with 33.1% fewer inquiry turns than GRPO.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, medical agents, multi-turn dialogue, credit assignment, privileged information, counterfactual learning, large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_NKXKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PCQC, an RL-based method to train the model to learn from both executed and unexecuted questions without completing a full rollout in multi-turn medical conversations. It adds question-level credit to GRPO for those useful questions. It reported an improvement over vanilla SFT, GRPO (with simple correctness reward), and ATPO.

Critical strengths

The method has a clear intuition: It compares alternative questions to learn what to ask. The experiment setup is clearly described.

Critical weaknesses
  1. I think adding a prompt-only baseline is necessary. The original model could simply be instructed to ask for missing information before answering for the multi-turn interaction.
  2. Figure 3 shows fewer turns, but this could come from asking for more facts at once. It is unclear whether PCQC uses this shortcut. Reporting the number of unique facts elicited per turn across methods would help assess this potential reward hack.
Questions for authors
  1. Were the original ATPO settings used? What explains the lower SFT and ATPO scores compared with the original paper?
  2. How diverse are the sampled questions? Do they request different information, or are they often paraphrases?
1 comment

We are sincerely grateful to the reviewer for the careful review and for the time and effort devoted to our paper, and we appreciate that the reviewer found the intuition and setup clear.

W1. Prompt-only baseline. We thank the reviewer for suggesting this baseline. We evaluated the original Qwen3-8B on MedQA with a prompt explicitly instructing it to ask for missing information before answering. It correctly answers 589 of 1,268 cases (46.45% accuracy), close to the 46.03% reported for interactive MEDIQ prompting in the original ATPO paper (Table 1).

MethodMedQA accuracy (%)
Prompt-only46.45
Shared SFT46.69
GRPO56.28 ± 0.66
ATPO56.70 ± 1.04
PCQC62.54 ± 1.64

Results for GRPO, ATPO, and PCQC are means and standard deviations over three independent RL training runs. PCQC improves accuracy by 16.09 percentage points over prompt-only, supporting the benefit of training the inquiry policy beyond explicitly prompting the model to request missing information. All RL methods start from the same Shared SFT checkpoint; PCQC further outperforms GRPO and ATPO by 6.26 and 5.84 percentage points, respectively, demonstrating its advantage from a common initialization.

W2. Facts per turn. In an analysis of archived dialogues, we count new patient-supported facts per non-final attempt, excluding initial information and counting repeats once. A Qwen3-8B labeler, blind to method and correctness, aligns replies to recorded facts. Labels are complete for all three methods on 383 of 2,093 cases so far; we will report the full cohort in the appendix:

PCQCGRPOATPO
New facts / attempt1.080.200.30
New facts / case1.660.470.32
Attempts / case1.532.341.08
Attempts with ≥2 new facts26%5%9%

PCQC obtains 5.4 times as many new facts per attempt as GRPO and 3.6 times as many as ATPO, and more than 3.5 times as many per case as either. Part of this comes from broader requests (88% of its first MedQA questions mention examination findings or tests), which we read as effective information acquisition, not a reward hack:

  1. No reward counts turns or facts: the terminal reward is correctness only, and question credit ranks same-state candidates by diagnostic utility. In MedQA state 702 from the Section 4.4 set, a narrow question on leakage triggers (short reply) received +0.83 credit, and a request returning seven facts received +0.79. A compound request whose reply added high fluid intake received −1.25: after it, the scorer's correct-option probability fell from 0.91 to 0.01.
  2. The responder is instructed to return only recorded facts, so broad requests gain nothing beyond the record.
  3. Stacked questions are rarer for PCQC: 6% of its MedQA questions contain more than one question mark (GRPO 70%, ATPO 73%).

Q1. ATPO settings. The algorithm settings are original: the official code and run-script settings at the released commit (U1+U2, M = 128, four candidates, threshold 1.2, four critic warmup updates, GAE with γ = λ = 1, +3/0/−1 reward; Appendix A.2). Where the ATPO paper's text differs (τ = 1.5, five warmup steps), we follow the script. Two inputs are not public:

  • No SFT trainer is released, so all methods start from our Shared SFT on the 4,157 released conversations (the ATPO paper describes 1,269).
  • The released code calls a patient service that is not public, so all methods use our frozen Qwen3-8B responder.

Both differences already apply at the SFT start, and ATPO's gain over that start is nearly the original one (ATPO paper Table 1, Qwen3-8B, same test sets; last column: mean over its three sets):

MedQASFTGRPOATPOATPO − SFT3-set ATPO − SFT
ATPO paper53.7557.9264.07+10.32+9.04
Ours46.6956.28 ± 0.6656.70 ± 1.04+10.01+9.11

The lower absolute score thus follows the lower start.

Q2. Diversity. The requested information usually differs, while phrasing often overlaps. Among the 512 candidates (128 MedQA states × 4) from Section 4.4:

  • States average 3.50 distinct question texts and 3.55 distinct answers out of 4. In 107/128 states, the answers return different sets of recorded facts (lexical matching).
  • In state 702 (W2), the four requests target different information, and the scorer's correct-option probability after the answers ranges from 0.009 to 0.989.
  • Phrasing often overlaps: 77% of candidates open with "What were the findings…".

Picking the highest-utility candidate raises downstream success by 4.20 points over uniform selection [1.42, 6.96].

Reviewer_Frv3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposed PCQC (Privileged Counterfactual Question Credit), which learns from questions never asked by using privileged patient facts to answer alternative questions at the same dialogue state. It is a reinforcement learning framework to improve information acquisition during multi-turn medical diagnostic dialogue. A frozen diagnostic scorer evaluates each resulting question–answer pair, and within-state comparisons yield relative question credit for both executed and unexecuted questions without rolling the alternatives forward. The method is evaluated using Qwen3-8B on MedQA, MedicalExam, MedMCQA, and iCRAFT-MD. It outperforms GRPO and ATPO.

Critical strengths
  1. Reasonable novelty - the proposed approach is novel and meaningful. The alternative actions are unexamined in traditional reinforcement learning.

  2. Strong experimental results and ablation studies. In addition to outperforming standard GRPO, the Same-State-Q1 isolates the effect of applying the loss to unexecuted candidates. Main methods are run three times. Downstream continuation experiment provides evidence that the diagnostic utility is correlated with downstream success.

Critical weaknesses
  1. The evaluation is done on fixed-choice questions rather than open-ended clinical interactions. MedQA is in-distribution with the RL training data. Only iCRAFT-MD is out-of-distribution.

  2. The scorer is larger than the policy. Some gain may come from distillation of the stronger model's judgement.

  3. The assumption of privileged patient facts can be transformed into reliable answers for questions never asked is questionable. This assumption doesn't work in real world clinical settings.

Questions for authors
  1. How to bridge the gap between the settings in the paper and the real world clinical situation? (as mentioned in weakness)
1 comment

We sincerely thank the reviewer for the careful and thorough review and for the time and effort devoted to our paper. We are grateful that the reviewer recognized our controls and continuation analysis.

W1. Fixed-choice evaluation and distribution. We will make the scope explicit: the doctor asks free-text questions and receives free-text answers, and every case ends in a fixed-choice decision. Our evidence covers information acquisition for exam-derived, fixed-choice decisions; open-ended consultation is outside this scope.

BenchmarkRelation to RL trainingAcc. PCQC / GRPO / ATPOQ/ep PCQC / GRPO
MedQADev set; same source as 9,400 training cases62.54 / 56.28 / 56.701.508 / 2.280
MedMCQAHeld-out split; same source as 4,856 training cases54.48 / 49.19 / 50.121.560 / 2.142
MedicalExamSeparate ATPO exam set; 0/150 exact case overlap61.78 / 61.56 / 61.111.549 / 2.358
iCRAFT-MDIndependent dermatology source; no lexical match found73.62 / 67.87 / 67.631.410 / 2.228

PCQC is more accurate than GRPO and ATPO on all four benchmarks and asks fewer questions than GRPO on each. On iCRAFT-MD, the only independently sourced benchmark, its gain is on par with MedQA's: +5.75 over GRPO and +5.99 over ATPO, with 36.7% fewer inquiries than GRPO. MedicalExam adds a format shift: 86 of its 150 cases have 5–8 options, while every training case has four. PCQC still leads there, with 34.3% fewer inquiries than GRPO. We will also note MedMCQA's shared training source.

W2. Larger scorer. Three results indicate that PCQC's advantage comes from how it uses the scorer's signal, and that a same-size scorer is equally predictive:

  • A same-size scorer is as predictive (additional experiment). We re-scored the 512 Section 4.4 candidates with official Qwen3-8B, the policy's base model, keeping prompts, readout, and continuations fixed. Selecting its highest-utility candidate improves downstream success by 4.59 points over uniform selection [1.88, 7.28], versus 4.20 [1.42, 6.96] with the 14B scorer.
  • Same scorer, different use (Table 3; same initialization, data, G_T = 4; three-benchmark means). Adding the scorer's credit for the executed question only (Executed-local) raises Terminal-only from 49.38% to 54.24%. PCQC, with the same responder, scorer, and candidates, applies same-state relative credit to all four candidates: 59.60 ± 1.25% (three runs). Using the same scorer this way adds 5.36 points, more than adding its feedback at all (+4.86). On MedQA, PCQC Run A − Executed-local is +4.42 points [1.26, 7.49], p = 0.0067 (Table 9).
  • Table 10, Same-State-Q1. Same candidate groups and same-state credit, but the loss covers only the executed question, normalized over its tokens: 59.95% versus 63.02% for PCQC Run A (four-benchmark means, one run each).

The policy never imitates scorer outputs: the scorer returns one probability per question–answer pair, standardized within its candidate group into a scalar credit. The scorer never observes the case record directly and is not used at evaluation. We will add the same-size analysis.

W3 and Q1. Privileged facts and clinical transfer. Only in training does the responder answer alternative questions from privileged facts, to compute credit. At deployment the policy sees only the dialogue, task, and options, with no case record, candidate construction, or scorer.

Answers need not be complete. The responder is told to use only recorded facts and otherwise reply "The patient cannot answer this question." (Appendix B.2). Such replies stay in the group and receive credit: in the 24 continuation states where refusals and answers mix, an answered candidate gets the highest credit in 21.

Answer reliability. A model-assisted audit with two judges (Qwen3-14B, Qwen3-8B) covered the 429 higher-risk continuation candidates. The 14B judge rates 399/409 non-abstaining answers as supported by the record; averaged over both judges, 1.5% reveal unrequested facts. Appendix D.4 keeps 395/512 candidates whose answers are supported, limited to the request, and free of decision-target disclosure. On these, highest-utility selection still improves downstream success by 3.52 points over uniform selection [1.06, 6.07] (Table 11).

Toward clinical settings (Q1). Incomplete records make the signal conservative: when the record lacks an answer, the responder abstains, so a question can be under-credited rather than rewarded for invented facts. Steps toward clinical use:

  1. Build fact inventories from de-identified clinical encounters, keeping the abstention rule.
  2. Check responder fidelity on a clinician-reviewed sample, or use recorded real dialogues.
  3. Replace the option-probability utility with an open-ended one, e.g., the likelihood of the reference diagnosis.
  4. Evaluate with clinicians or standardized patients before any clinical use, consistent with our ethics statement.

We will add this roadmap.

Reviewer_3gbXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Goal: To improve RL-based medical dialogue agents.

Details:
The paper considers standard RL-based training of a medical dialogue agent where the agent interacts with a patient simulator and concludes the session with diagnosis. Such unrolling is treated as a RL trajectory where the actions are inquiries made by the agent and a reward is assigned, usually at the end of the session (e.g. diagnostic evaluation).
This approach however is not efficient, since only a limited number of branches can be explored from a given point in the simulated dialogue/trajectory. The paper challenges this limitation and asks whether questions that are not unrolled during simulation can also be used for the training. The paper then proposes PCQC (privileged counterfactual question credit). Unlike standard RL, PCQC uses privilege information from the patient notes to assign credit to the queries that are generated but not used during simulations thus improving the efficiency. This privileged information is only used at the training time. The results across 4 dataset show that PCQC outperforms RL methods such as GRPO and ATPO by substantial margins.

Critical strengths

I love the premise of the paper. RL training is generally hindered by the number of roll-outs involved in the simulation. As the depth of the simulation increases, so does the branching of the roll-outs. The proposed approach instead uses insight from the medical domain and leverages hidden patient information to assign credits to the unrealised branches. Section 3 lays out the foundation and mathematics clearly and connects the choice of the scorer to question utility. I find this exciting and novel.

The paper also compares training efficiency of PCQC against GRPO and shows that PCQC provides gains with the same or less GPU hours.

Critical weaknesses
  1. Although the paper reports quantitative gains, it lacks any qualitative evaluation. Have authors conducted any human evaluation of the responses from the PCQC system and baselines? If reasonable, I suggest they do so. Nonetheless, I expect some representative examples that showcase improvements of PCQC over RL baselines and provide qualitative insights.
  2. I am a little skeptical of results reported in table 1 and table 2. For all the baselines, number turns is less than 3, i.e., the rollouts or trajectories are very short. I don’t think this is expected unless either most of the patient information is made available to the agent at the start and the agent can form a reasonable diagnosis without requiring much patient clarification or RL training somehow hacks the terminal rewards. Can authors provide some insights here?
  3. It is unclear from the paper whether reasoning is enabled for the PCQC model and the baselines. RL methods such as GRPO were designed for allowing models to think before they answer. Does the PCQC model think differently compared to baselines?
Questions for authors

I like the way paper is written and experiments are conducted. I request authors address my concerns from the weakness section.

1 comment

We sincerely thank the reviewer for the careful reading of our paper and for the time and effort devoted to reviewing it. We are grateful for the encouraging comments on the premise, Section 3, and training efficiency, and we answer the three weaknesses in turn.

W1. Qualitative evidence. Below is a qualitative analysis of archived evaluation dialogues and a full-run statistic; we will add both to the appendix (we did not run a formal human evaluation).

MedQA case 450 (quoted text is verbatim; reasoning omitted). Initial information: "A 63-year-old man is aiming to improve his health by eating a well balanced diet, walking daily, and quitting smoking following a 45-year smoking history." Problem: "What is the mechanism of action of this medication?" Gold: D (phosphodiesterase inhibitor). The hidden record states that cilostazol is prescribed; the initial information never names the drug.

  • PCQC (1 inquiry, correct). "What medication was the patient prescribed?" → "Cilostazol." → Final Answer: D.
  • GRPO (3 inquiries, correct). Inquiry 1 asks about respiratory symptoms "since he started these new health habits"; inquiry 2 asks about "a recent medical check-up or any cardiac evaluations, like an echocardiogram". Both receive "The patient cannot answer this question." Inquiry 3 asks for "the drug's known effect"; the reply names cilostazol, and the doctor answers D.

The missing evidence is the drug's identity. PCQC asks for it directly; GRPO first follows background cues that the record cannot answer. Across five decoding seeds for this case, PCQC is correct 5/5 times with one inquiry each time; GRPO and ATPO are correct 3/5 times.

Full-run statistic. Share of actual patient replies that are exactly "The patient cannot answer this question." (the responder's fixed reply when no recorded fact answers; format-error feedback excluded), one complete evaluation run per method:

BenchmarkPCQCGRPOATPO
MedQA29.7%56.3%44.1%
MedicalExam35.2%65.9%56.6%
MedMCQA51.5%78.8%69.8%
iCRAFT-MD20.9%47.5%34.4%

On every benchmark, PCQC's questions are refused least often: 26.6–30.7 points less than GRPO's and 13.5–21.4 less than ATPO's. Without stacked questions (one question mark only), the gaps remain 23.7–48.9 and 16.1–39.2 points: PCQC more often asks for what the record holds.

W2. Short trajectories. The agent sees little of the case at the start, and the evidence points to efficient inquiry rather than reward hacking:

  • Initial information is brief. The first turn gives a vignette of 13–20 words on average (by benchmark), plus the question and options. By a lexical-overlap approximation, 3.1–12.5 recorded facts per case remain hidden on average. The doctor prompt states: "The initial patient information provided by the user in the first round is incomplete."
  • The options narrow the decision. One or two targeted facts often settle a case, as with the drug name above.
  • Metric. Q/ep counts non-final attempts and excludes the final answer; a consultation allows at most 10 actions.
  • Short dialogues predate RL. Shared SFT already averages 1.460 inquiries before RL; GRPO raises this to 2.252 (Table 2). GRPO and PCQC use a correctness-only terminal reward, with no bonus for fewer turns.
  • No test-time reward to exploit. Accuracy is scored against reference answers; the diagnostic scorer is not used at evaluation. PCQC gains 13.56 points over Shared SFT at nearly the same length (1.460 vs. 1.507).
  • Little answer leakage. Replies to PCQC rarely contain the correct option's text: 0.5% of informative replies on MedQA, 0.6% on iCRAFT-MD, none on MedicalExam or MedMCQA.

PCQC also obtains more new recorded facts per inquiry: 1.08 versus 0.20 for GRPO and 0.30 for ATPO on the 383 cases labeled so far for all three methods (model-assisted labels).

W3. Reasoning. Thinking mode is enabled for every doctor policy (Shared SFT, GRPO, ATPO, PCQC), in RL rollouts and in evaluation. Decoding is identical: temperature 1.0, top-p 0.8, and 512 tokens per action covering reasoning and the question or final answer (Appendix A.3); we will also state it in Section 4.1. In archived outputs on all four benchmarks, all 2,093 PCQC and ATPO episodes and 2,083/2,093 GRPO episodes contain a non-empty reasoning block.

The training signal differs. PCQC's reasoning tokens receive only the terminal GRPO credit, as in GRPO; the question-level loss covers only the question text. Inference uses no scorer or candidate search. The outputs differ observably. On MedQA, PCQC generates 313.7 tokens per episode, reasoning included, versus 473.0 for GRPO (Table 8); reasoning is 84% of PCQC's tokens in the archived run (GRPO 63%), so it saves mostly on question text. 88% of its first questions contain examination or test terms (GRPO 25%, ATPO 25%), and 6% of all its questions contain more than one question mark (GRPO 70%, ATPO 73%).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.