- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces PCQC, an RL-based method to train the model to learn from both executed and unexecuted questions without completing a full rollout in multi-turn medical conversations. It adds question-level credit to GRPO for those useful questions. It reported an improvement over vanilla SFT, GRPO (with simple correctness reward), and ATPO.
Critical strengths
The method has a clear intuition: It compares alternative questions to learn what to ask. The experiment setup is clearly described.
Critical weaknesses
- I think adding a prompt-only baseline is necessary. The original model could simply be instructed to ask for missing information before answering for the multi-turn interaction.
- Figure 3 shows fewer turns, but this could come from asking for more facts at once. It is unclear whether PCQC uses this shortcut. Reporting the number of unique facts elicited per turn across methods would help assess this potential reward hack.
Questions for authors
- Were the original ATPO settings used? What explains the lower SFT and ATPO scores compared with the original paper?
- How diverse are the sampled questions? Do they request different information, or are they often paraphrases?
We are sincerely grateful to the reviewer for the careful review and for the time and effort devoted to our paper, and we appreciate that the reviewer found the intuition and setup clear.
W1. Prompt-only baseline. We thank the reviewer for suggesting this baseline. We evaluated the original Qwen3-8B on MedQA with a prompt explicitly instructing it to ask for missing information before answering. It correctly answers 589 of 1,268 cases (46.45% accuracy), close to the 46.03% reported for interactive MEDIQ prompting in the original ATPO paper (Table 1).
Results for GRPO, ATPO, and PCQC are means and standard deviations over three independent RL training runs. PCQC improves accuracy by 16.09 percentage points over prompt-only, supporting the benefit of training the inquiry policy beyond explicitly prompting the model to request missing information. All RL methods start from the same Shared SFT checkpoint; PCQC further outperforms GRPO and ATPO by 6.26 and 5.84 percentage points, respectively, demonstrating its advantage from a common initialization.
W2. Facts per turn. In an analysis of archived dialogues, we count new patient-supported facts per non-final attempt, excluding initial information and counting repeats once. A Qwen3-8B labeler, blind to method and correctness, aligns replies to recorded facts. Labels are complete for all three methods on 383 of 2,093 cases so far; we will report the full cohort in the appendix:
PCQC obtains 5.4 times as many new facts per attempt as GRPO and 3.6 times as many as ATPO, and more than 3.5 times as many per case as either. Part of this comes from broader requests (88% of its first MedQA questions mention examination findings or tests), which we read as effective information acquisition, not a reward hack:
Q1. ATPO settings. The algorithm settings are original: the official code and run-script settings at the released commit (U1+U2, M = 128, four candidates, threshold 1.2, four critic warmup updates, GAE with γ = λ = 1, +3/0/−1 reward; Appendix A.2). Where the ATPO paper's text differs (τ = 1.5, five warmup steps), we follow the script. Two inputs are not public:
Both differences already apply at the SFT start, and ATPO's gain over that start is nearly the original one (ATPO paper Table 1, Qwen3-8B, same test sets; last column: mean over its three sets):
The lower absolute score thus follows the lower start.
Q2. Diversity. The requested information usually differs, while phrasing often overlaps. Among the 512 candidates (128 MedQA states × 4) from Section 4.4:
Picking the highest-utility candidate raises downstream success by 4.20 points over uniform selection [1.42, 6.96].