acceptodds
Under review as a conference paper at ICLR 2027

MedPreceptor: Decoupled Process Supervision for Active Clinical Inquiry in Multi-Turn Medical Consultations

Abstract

Effective medical consultation requires not only diagnostic reasoning over existing evidence, but also actively eliciting crucial clinical information across multi-turn interactions. However, both human clinicians and Medical LLMs frequently exhibit process-level drawbacks during clinical inquiry, such as repetitive questioning, misdirected inquiry, omission of key evidence, and premature diagnostic closure. Existing methods typically address this by fine-tuning the Doctor LLM itself, which tightly couples consultation execution with process supervision and impedes reusability across heterogeneous models and human clinicians. To address this limitation, we introduce MedPreceptor, a decoupled process supervision framework that selectively intervenes in multi-turn consultations while leaving the underlying Doctor LLM unchanged. To ensure robust decision-making, we construct prefix-aligned supervision via a two-stage annotation pipeline that eliminates future information leakage, and develop Decision-Guided Selective On-Policy Distillation (DG-OPD). By selectively routing teacher signals conditioned on intervention decisions, DG-OPD trains the supervisor on trajectories from three Doctor LLMs alongside real doctor-patient dialogues, enabling it to learn both when to intervene and what actionable guidance to provide. In addition, we construct the Clinically Annotated Consultation Intervention Benchmark (CACI), comprising 1,664 expert-annotated consultation states for evaluating intervention timing and the quality of corrective advice. Across five Doctor LLMs on MAQuE, MedPreceptor improves diagnostic accuracy by up to 10.27%; across seven Doctor LLMs on MedDialogRubrics, it increases clinically important information coverage by up to 7.08 %. On CACI, MedPreceptor achieves 66.11% accuracy, 67.70% F1, and a corrective advice quality score of 7.71/10. These results demonstrate that decoupling process supervision from consultation execution offers an effective, generalizable paradigm for advancing active clinical inquiry across diverse medical LLMs. Code is available at https://anonymous.4open.science/r/MedPreceptor-5E1F.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Medical LLMs, Multi-Turn Medical Dialogue, Clinical Information Gathering, Process Supervision, On-Policy Distillation

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_wEVUOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an independent supervision model that monitors doctor–patient conversations and offers guidance when the doctor model repeats questions, misses clues, or reaches conclusions prematurely. Contributions include a plug-in selective supervision framework, a two-stage annotation process that avoids using future information in final supervision, and Decision-Guided Selective On-Policy Distillation (DG-OPD). The key design is to retain normal teacher guidance when the student’s intervention decision is correct, but retain only suppressive signals when it is incorrect, avoiding reinforcement of an erroneous decision. The authors generate consultation trajectories from multiple doctor models, use complete conversations to identify candidate intervention points, and then reassess those points using only the information available at that moment. Teacher and student supervisors are trained to output an intervention decision, supporting evidence, and corrective advice. During distillation, the learning signal depends on whether the intervention decision is correct. At deployment, the supervisor reads the ongoing conversation and provides next-step guidance when needed.

Critical strengths
  • This paper offers a practically useful modular design. The supervisor supports multiple open- and closed-source doctor models without modifying their parameters, with relatively consistent improvements across models.
  • Aligns methodological choices with clearly defined challenges. Two-stage annotation restricts future information from entering supervision, DG-OPD adjusts learning signals based on intervention correctness, and CACI separately evaluates intervention capability.
Critical weaknesses
  • The clinical reliability of generated advice lacks independent expert validation. Human annotations in CACI support intervention-decision evaluation, but advice quality is primarily judged by GPT-5.6-Sol, without reported agreement with clinical experts. Higher automated scores therefore do not establish that the advice is safe, appropriate, or beneficial to subsequent consultation. Authors should conduct blinded clinical expert evaluation of representative outputs, including unnecessary interventions, assessing correctness, necessity, actionability, and potential harm. Report inter-expert agreement and agreement between experts and the LLM judge.
  • Efficiency gains are inconsistent, and budget-matched comparisons are missing. For GPT-5.6-Sol, average consultation length increases from 5.69 to 7.38 turns, while satisfied rubrics per turn decrease from 1.234 to 1.029, suggesting that higher coverage may partly result from longer conversations. Although other models show improved per-turn efficiency, the additional system-level overhead remains unclear. Compare performance under fixed dialogue-turn and total-token budgets, including supervisor calls, and report coverage, latency, monetary cost, and information gained per unit cost.
Questions for authors
  • Appendix B.3 instructs annotators not to label alternative but clinically reasonable inquiry strategies as errors. What was the agreement among the three CACI annotators before adjudication? Could the authors provide agreement statistics and representative borderline cases showing how this principle was applied consistently?
  • Although Stage B uses only the current dialogue prefix, candidate intervention points are selected by Stage A with access to the complete conversation and final diagnosis. Have the authors assessed whether this selection process misses problems identifiable from the prefix alone or disproportionately selects particular error types?
6 comments

We thank the reviewer for the insightful review and constructive feedback. We conducted additional experiments and analyses to address the following questions:

Q1: How reliable is the generated advice when evaluated by blinded clinical experts, and how well does the LLM judge align with human raters?

Response to Q1: Blinded Clinical Expert Evaluation of Advice

1. Evaluation setup. We conducted a blinded evaluation across 64 representative consultation prefixes, including both necessary and unnecessary intervention states, where six models each generated guidance, yielding 384 anonymized suggestions. Two clinical experts and an LLM judge (GPT-5.6-Sol) independently scored each suggestion solely from the visible case context and dialogue prefix (without access to reference advice or model identity) across six dimensions from 0 to 2, for a maximum total score of 12. The score dimensions are Problem Alignment, Clinical Actionability, Grounding, Efficiency and Non-redundancy, Advisor Style and Safety, and Intervention Necessity.

2. Inter-expert and LLM-judge agreement. Table 1 reports unweighted (Gwet's AC1) and ordinal-weighted (Gwet's AC2) agreement, which scale from -1 (systematic disagreement) through 0 (chance) to +1 (perfect agreement). The two clinical experts show high consistency across all dimensions, particularly on Clinical Actionability (0.958), Grounding (0.953), and Intervention Necessity (0.959). The LLM judge aligns closely with both clinicians on core quality dimensions (Problem Alignment, Clinical Actionability, Grounding, and Advisor Style / Safety), validating its reliability for large-scale evaluation. Agreement on Efficiency / Non-redundancy is lower both between the LLM and experts and between the two clinicians themselves, possibly because different experts may adopt different yet reasonable inquiry strategies for the same case.

Table 1. Pairwise Agreement on Blinded Advice Ratings (Gwet's AC1 / AC2).

Evaluation DimensionExpert 1--Expert 2Expert 1--LLMExpert 2--LLM
Problem Alignment0.610 / 0.8800.509 / 0.8280.557 / 0.844
Clinical Actionability0.843 / 0.9580.834 / 0.9550.899 / 0.972
Grounding0.873 / 0.9530.781 / 0.9170.870 / 0.957
Efficiency / Non-redundancy0.488 / 0.8130.089 / 0.3280.240 / 0.499
Advisor Style / Safety0.744 / 0.9140.720 / 0.8740.779 / 0.911
Intervention Necessity0.850 / 0.9590.522 / 0.7730.541 / 0.793

3. Advice quality and calibrated safety assessment. As shown in Table 2, MedPreceptor receives average scores of 11.17 and 11.72 from the two clinical experts, compared with 11.14 and 11.80 for GPT-5.6-Sol. Neither clinical expert nor the LLM judge flagged any suggestion in these samples as introducing direct clinical harm or unsafe patient guidance (0/384). While this indicates that prefix-grounded process reminders rarely introduce overt safety violations in the evaluated states, it does not guarantee error-free deployment: as analyzed in Appendix B.4, suboptimal question prioritization or unsupported diagnostic assumptions can still occur, and we have revised our manuscript claims accordingly.

Table 2. Blinded Advice-Quality Scores Across Evaluators.

Evaluated ModelExpert 1Expert 2LLM Judge
MedPreceptor11.1711.7210.78
GPT-5.6-Sol11.1411.8011.53
Baichuan-M2-32B10.9511.029.08
Doctor-R110.7710.488.09
DeepSeek-V4.1-Flash11.0011.679.84
Gemma-4-31B-IT10.8410.849.94

Q2: Does MedPreceptor remain effective under matched turn and token budgets, and what is its system-level overhead?

Response to Q2: Budget-Matched Efficiency and System Overhead

We conducted fixed total-token and fixed maximum-turn comparisons on MedDialogRubrics. We report coverage, consultation length, decomposed Doctor and Preceptor token usage, supervisor call frequency, end-to-end latency, and information gained per Doctor token. We report Rubrics / 1K Doctor Tokens because MedPreceptor can be deployed locally at relatively low cost, whereas Doctor LLMs are typically much more expensive to run. This metric therefore emphasizes information-gathering efficiency relative to the dominant inference cost.

1. Fixed Total-Token Budget. We evaluated GPT-5.6-Sol across 1,600 MedDialogRubrics cases under an externally enforced 6K total-token cap per case that counts both Doctor and Preceptor input/output tokens together. As shown in Table 3, MedPreceptor increases coverage from 54.48% to 56.37%, while reducing average consultation length from 3.5184 to 3.4084 turns. Satisfied rubrics per turn also increase from 2.033 to 2.175.

Table 3. Fixed 6K Total-Token Budget on MedDialogRubrics (GPT-5.6-Sol, 1,600 cases).

PreceptorMacro CoverageAvg. QA TurnsDoctor Tokens /CasePreceptor Tokens /CasePreceptor Calls/CaseRubrics /TurnRubrics / 1K Doctor Tokens
No54.48%3.51843006.770.000.00002.0332.379
Yes56.37%3.40842745.591315.980.79492.1752.700

2. Fixed Maximum-Turn Budget. We further compared GPT-5.6-Sol (2,000 cases, Table 4) and Baichuan-M2-32B (Table 5) under matched maximum-turn caps. Across the three turn limits (Maxturn = 6,10,12), MedPreceptor improves rubric coverage by 5.71, 3.52, and 3.27 percentage points on GPT-5.6-Sol, and by 0.10%, 4.90%, and 7.08% on Baichuan-M2-32B. Average Preceptor calls per case across the three turn caps in Table 4 are 0.8430, 1.0745, and 1.6720.

Table 4. Fixed Maximum-Turn Budget on MedDialogRubrics (GPT-5.6-Sol, 2,000 cases).

MaxturnPreceptorCoverageAvg. QA TurnsDoctor Tokens /CasePreceptor Tokens /CasePreceptor Calls/CaseRubrics / 1K Doctor Tokens
6--49.83%3.91003444.080.000.00001.921
6✓55.54%4.06703570.491387.810.84302.065
10--53.50%4.00953950.230.000.00001.798
10✓57.02%4.31904361.621917.431.07451.736
12--53.81%5.22354497.250.000.00001.588
12✓57.08%6.43656509.243004.601.67201.164

3. Latency and cost. To avoid network variance from external commercial APIs, we measured end-to-end consultation wall-clock time on locally deployed Baichuan-M2-32B, including all supervisor checks (Table 5). At Maxturn = 6, 10, and 12, MedPreceptor adds 1.2, 3.5, and 4.2 seconds of end-to-end consultation time per case, respectively, while clinical coverage substantially expands. We also report Rubrics / 1K Doctor Tokens, the number of satisfied clinical rubrics per 1,000 Doctor input/output tokens. This measures efficiency relative to Doctor token usage. A single monetary-cost figure depends on the API pricing or local hardware used, so we report the measured token and time costs instead of assuming a universal dollar cost.

Table 5. Fixed Maximum-Turn Budget and Latency on MedDialogRubrics (Baichuan-M2-32B).

MaxturnPreceptorCoverageAvg. QA TurnsDoctor Tokens /CasePreceptor Tokens /CasePreceptor Calls/CaseRubrics / 1K Doctor TokensTime (s)
6--32.19%5.79184500.002.33917.7
6✓32.29%5.79186312781.002.32418.9
10--32.22%6.68277200.001.55920.5
10✓37.12%6.99300926971.911.65624.0
12--32.65%6.94294200.001.49321.1
12✓39.73%7.30328628502.011.62725.3

Q3: What is the pre-reconciliation inter-annotator agreement in CACI, and how are clinically reasonable borderline cases handled?

Response to Q3: CACI Annotation Agreement and Borderline Cases

We report an additional reconciliation review by three human experts.

1. Pre-reconciliation decision agreement. Before adjudication, all three clinical experts unanimously agreed on 82.93% of the CACI records (multi-rater Gwet's AC1 = 0.782), while remaining 17.07% exhibited a 2:1 split resolved by majority vote. Pairwise agreement across the benchmark is summarized in Table 6.

Table 6. Pairwise Inter-Expert Agreement on CACI Intervention Decisions.

ComparisonAgreement (%)Disagreement (%)Cohen's κGwet's AC1
Expert 1--285.0414.960.6790.719
Expert 1--390.879.130.8130.822
Expert 2--389.9610.040.7940.806

2. Advice reconciliation and topic consistency. When the majority voted to intervene, we compared the advice-topic label sets from all experts supporting intervention. If all sets matched, one supporting suggestion was selected at random; if any set differed, the experts discussed the case to reach a final suggestion. Under this protocol, only 7.87% of CACI records required advice reconciliation, with high inter-expert agreement across individual clinical advice topics (Table 7).

Table 7. Per-Topic Inter-Expert Agreement on Advice Content (Gwet's AC1).

Advice TopicExpert 1–Expert 2Expert 1–Expert 3Expert 2–Expert 3Three-Expert AC1
Medication dose / timing0.9540.9720.9790.968
Repeated questioning0.7580.8530.8760.830
Abdominal pain / vomiting / bowel0.9900.9901.0000.993
Triggers and symptoms0.9570.9760.9750.969
Nonexclusive diagnosis1.000*1.000*1.000*1.000*
Relevant medical history0.9360.9640.9670.956
Other0.8690.9490.8990.906

* No positive instances occurred for the nonexclusive-diagnosis topic; the value 1.000 reflects unanimous absence, not agreement on detecting that topic.

3. Agreement across clinical specialties. Pre-reconciliation decision agreement also remains consistently strong across all six clinical specialties in CACI (Table 8). The three-expert unanimous agreement ranges from 76.03% to 90.22%, with an overall rate of 82.93%. Multi-rater Gwet's AC1 ranges from 0.688 to 0.888 across specialties.

Table 8. Pre-Reconciliation Decision Agreement by Clinical Specialty.

Clinical SpecialtyThree-ExpertExpert 1--2Expert 1--3Expert 2--3
Respiratory Medicine83.33% / 0.78083.33% / 0.67683.33% / 0.676100.00% / 1.000
Cardiology76.03% / 0.68880.58% / 0.62788.43% / 0.77283.06% / 0.669
General Internal Medicine80.51% / 0.74383.90% / 0.68689.41% / 0.78887.71% / 0.758
Gastroenterology90.22% / 0.88891.67% / 0.86095.65% / 0.92393.12% / 0.881
Hematology80.24% / 0.73784.98% / 0.70090.12% / 0.80485.38% / 0.708
Rheumatology and Immunology85.01% / 0.82585.01% / 0.74695.61% / 0.92089.41% / 0.815
Overall82.93% / 0.78285.04% / 0.71990.87% / 0.82289.96% / 0.806

Each cell in Table 8 reports raw agreement rate followed by Gwet's AC1 (Three-Expert denotes unanimous agreement and multi-rater AC1; remaining columns report pairwise agreement).

4. Representative borderline cases. The following two 2:1 split cases illustrate how our protocol preserves alternative yet clinically reasonable inquiry strategies without misclassifying them as errors:

  • Case 1 (Cardiovascular----symptom refinement vs. repetition): An 85-year-old man with coronary artery disease and a prior bare-metal stent presented with worsening exertional chest pain and dyspnea. After asking about shortness of breath at Turn 7 (eliciting exertional dyspnea), the Doctor asked about chest tightness and breathlessness at Turn 10; while Expert 2 flagged this as repetitive (INTERVENE), Experts 1 and 3 judged it a clinically valid differentiation of cardiopulmonary symptoms, yielding a majority decision of NO_INTERVENE.

  • Case 2 (Hematology----competing clinical priorities): A 62-year-old man with celiac disease and prior gastrointestinal anastomoses presented with severe anemia (hematocrit 20), a positive fecal occult blood test, and unspecified discomfort following accidental gluten exposure. At Turn 4, the Doctor prioritized questions about possible gastrointestinal bleeding, including melena and hematochezia. Although Expert 1 preferred prioritizing the gluten trigger (INTERVENE), Experts 2 and 3 judged ruling out acute blood loss an appropriate clinical priority, resulting in a majority decision NO_INTERVENE.

Reviewer_hodyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MedPreceptor, a supervisor monitoring multi-turn medical consultations and advises a doctor LLM. It consists of two-stage annotation pipeline, first identifying candidate intervention retrospectively and adjudicating them using the visible dialogue prefix, then running Decision-Guided Selective On-Policy Distillation which removes positive distillation signals when the student’s intervention decision is incorrect. Evaluation reports diagnostic accuracy improvements across five doctor models on MAQuE, and coverage improvements on MedDialogRubrics.

Critical strengths

MedPreceptor shows that consultation supervision can transfer without retraining each doctor model, and can benefit both diagnostic accuracy and information coverage across different benchmarks.

This study showing the failure mode of naive on-policy distillation for the for decision-plus-generation outputs. DG-OPD improves CACI F1 and downstream MAQuE results.

Critical weaknesses
  • Compared methods focus on excluing supervisor or using different supervisor training methods. Some simple but meaningful baselines could be considered. Always intervene and fixed frequency advice baselines can help establish whether learning when to intervene provides benefits beyond routine guidance.

  • The two largest improvements in MAQuE Doctor Accuracy are for Baichuan-M2 and Doctor-R1, which generate trajectories used to train the supervisor. Improvements for doctor models not included from trajectory generation are smaller: +2.65% for GPT-5.6-Sol, +5.17% for DeepSeek, and +5.24% for GLM.The abstract’s “up to 10.27 points” statement therefore reflects the in-distribution setting and need to be clarified.

  • It is not fully clear whether improvement come from receiving an extra prompting. MedDialogRubrics results show that consultation length increases for most doctor models. The question-count control is only reported for Baichuan, not for the model see most increase in turns like GPT-5.6-Sol. Generic reminder and matched turn budget controls can strengthen the attribution of gains to learned guidance.

  • CACI benchmark uses independent annotation, but does not report inter-annotator agreement status. Annotation discrepancy information is needed to show how consistently annotators identify errors.

  • Figure 1 arrangement can be improved, the middle panel shows a lot empty space.

Questions for authors

How does MedPreceptor compare with always-intervene and fixed-frequency advice policies? Does selective intervention improve outcomes or reduce unnecessary guidance?

Can you distinguish results for training-source doctor models from those for unseen doctor models? Do the differences in gains persist after accounting for baseline performance and statistical uncertainty?

Do the gains persist under matched consultation-turn budgets?

What was the inter-annotator agreement on CACI intervention decisions before reconciliation, and how many of them require reconciliation?

6 comments

We thank the reviewer for the constructive feedback. We have conducted new experiments and statistical analyses to address each point below:

Q1: Does learned selective intervention outperform always-intervene and fixed-frequency advice policies?

Responses to Q1: Selective vs. Always-Intervene/Fixed-Frequency Interventions

We conducted additional experiments to evaluate whether selective interventions perform better than fixed-frequency interventions.

  1. Experimental setup. MedPreceptor evaluates whether intervention is needed every three turns. We compare it against baselines that unconditionally inject advice every 3 or 6 turns on MAQuE (Baichuan-M2-32B, 16-turn cap).

  2. Diagnostic performance. As shown in Table 1, MedPreceptor achieves 60.72% diagnostic accuracy, outperforming the every-3-turn and every-6-turn baselines by 6.24% and 8.03%, respectively. It also achieves higher information coverage (0.3618 vs. 0.3549 and 0.3415). The every-3-turn baseline always intervenes at every scheduled supervisor checkpoint. Both fixed-frequency baselines use the same advice-generation procedure as MedPreceptor, differing only in the intervention policy.

  3. Consultation efficiency. MedPreceptor reduces the proportion of consultations reaching the 16-turn limit to 55.24%, compared with 66.17% and 64.21% for the fixed-frequency baselines and 70.83% without supervision. Combined with its higher diagnostic accuracy and information coverage, this suggests that selective intervention promotes more efficient clinical inquiry and reduces unnecessary or redundant questioning.

These results show that selective intervention improves diagnostic accuracy and information coverage while reducing the proportion of consultations reaching the maximum turn limit.

Table 1. Comparison of Intervention Strategies on MAQuE (Baichuan-M2-32B). Hit Limit is the percentage of consultations reaching the maximum 16 turns.

Intervention StrategyDoctor Acc.GPT Acc.CoverageRelevanceAdherenceCoherenceClarityEmpathyHit Limit (%)
No MedPreceptor50.45%66.34%0.31300.87900.84600.57600.68200.404070.83
Fixed Every 3 Turns54.48%67.51%0.35490.88200.88190.65460.67840.408866.17
Fixed Every 6 Turns52.69%67.58%0.34150.87060.86600.62370.68210.408664.21
MedPreceptor60.72%68.24%0.36180.90080.87520.65030.68170.409455.24

Q2: How well does MedPreceptor generalize to Doctor models that are not used to generate training trajectories, and how statistically reliable are these gains?

Responses to Q2: Training-Source vs. Other Doctor Models

We have revised the manuscript to distinguish gains on training-source Doctor models from those on unseen models, and computed paired bootstrap 95% confidence intervals to quantify statistical uncertainty.

  1. Training-source Doctors. On MAQuE, MedPreceptor improves diagnostic accuracy by 10.27% for Baichuan-M2-32B and 8.79% for Doctor-R1.

  2. Unseen Doctors. For GPT-5.6-Sol, DeepSeek-V4.1-Flash, and GLM-5.3-Flash, which were not used to generate training trajectories, diagnostic accuracy improves by 2.65, 5.17, and 5.24 percentage points, respectively. On MedDialogRubrics, all five unseen Doctors also show information coverage gains of 4.24--5.03%.

  3. Statistical significance. As shown in Table 2, we computed paired bootstrap 95% confidence intervals for the MAQuE accuracy improvements. All five intervals exclude zero, providing statistical evidence of improvements for both training-source and unseen Doctors.

Overall, these results demonstrate that MedPreceptor consistently improves performance across different Doctor LLMs, including unseen models, with statistically supported gains within each model, supporting its generalization ability. We will clarify in the abstract that the largest gain of 10.27 percentage points is obtained on a training-source Doctor.

Table 2. MAQuE Doctor Accuracy With and Without MedPreceptor, With Paired Bootstrap 95% Confidence Intervals for Improvements.

DoctorTraining-Source DoctorBaseline Acc. (%)With MedPreceptor Acc. (%)Improvement (pp)95% CI (pp)
Baichuan-M2-32BYes50.4560.72+10.27[+8.79, +11.76]
Doctor-R1Yes46.0054.79+8.79[+7.34, +10.24]
GPT-5.6-SolNo71.9774.62+2.65[+1.03, +4.28]
DeepSeek-v4.1-flashNo56.9762.14+5.17[+4.38, +6.00]
GLM-5.3-FlashNo68.5973.83+5.24[+3.73, +6.76]

Baseline denotes the Doctor without MedPreceptor. Training-source Doctors were used to generate supervisor-training trajectories.

Q3: Are the improvements from tailored advice rather than generic prompting or more consultation turns?

Responses to Q3: Situation-Specific Advice and Matched Turn Budgets

To verify that gains stem from tailored clinical guidance rather than extra prompting or longer dialogues, we evaluated a Generic Reminder baseline and matched maximum-turn budgets.

1. Situation-Specific Advice vs. generic reminder. To evaluate the gain of situation-specific advice, we kept MedPreceptor’s intervention policy and reminder delivery unchanged, but replaced its tailored advice with a generic message such as "Review the conversation and reconsider your next question." We compare the performance of these two preceptor approaches with different Doctor models on the MAQuE dataset, as shown in Table 3

  1. Diagnostic accuracy: Situation-specific advice improves accuracy by 9.10%, 4.62%, and 0.69%, across Baichuan-M2-32B, GLM-5.3-Flash, and GPT-5.6-Sol, respectively, compared to generic reminders.
  2. Coverage and efficiency: Situation-specific advice increases clinical fact coverage on both Baichuan-M2-32B (+0.0329) and GLM-5.3-Flash (+0.0280) while requiring fewer dialogue turns across all three models. On GPT-5.6-Sol, the generic message achieves higher coverage, but situation-specific advice reaches higher diagnostic accuracy and relevance in nearly half the turns (8.601 vs. 4.919).

Table 3. Generic Reminder vs. Situation-Specific Advice on MAQuE.

DoctorAdvicesDoc. Acc.GPT Acc.Cov.Rel.Adh.Coh.Clar.Emp.Turns
Baichuan-M2-32BGeneric51.6266.410.32890.87100.85610.60900.68680.407314.947
Baichuan-M2-32BSituation-specific60.7268.240.36180.90080.87520.65030.68170.409414.720
GLM-5.3-FlashGeneric69.2170.790.36700.87330.98070.86630.81570.534612.965
GLM-5.3-FlashSituation-specific73.8373.000.39500.87300.98700.87600.79800.501012.088
GPT-5.6-SolGeneric73.9373.930.30550.89970.98710.87440.77800.40948.601
GPT-5.6-SolSituation-specific74.6274.620.23300.93700.96600.87400.61300.39404.919

2. Fixed maximum consultation turns. We evaluated Baichuan-M2-32B and GPT-5.6-Sol on MedDialogRubrics with the same maximum turn limits (6, 10, or 12) for each paired setting. At Maxturn = 6, GPT-5.6-Sol coverage improves from 49.83% to 55.54%, while average QA turns rise only from 3.9100 to 4.0670. At Maxturn = 12, Baichuan-M2-32B coverage improves from 32.65% to 39.73%, while average QA turns rise from 6.94 to 7.30. Under the same maximum consultation-turn limits, MedPreceptor still achieves substantial improvements in clinical information coverage.

Table 4. Fixed Maximum-Turn Budget on MedDialogRubrics (GPT-5.6-Sol, 2,000 cases).

MaxturnPreceptorCoverageAvg. QA TurnsDoctor Tokens /CasePreceptor Tokens /CasePreceptor Calls/CaseRubrics / 1K Doctor Tokens
6--49.83%3.91003444.080.000.00001.921
6✓55.54%4.06703570.491387.810.84302.065
10--53.50%4.00953950.230.000.00001.798
10✓57.02%4.31904361.621917.431.07451.736
12--53.81%5.22354497.250.000.00001.588
12✓57.08%6.43656509.243004.601.67201.164

Table 5. Fixed Maximum-Turn Budget on MedDialogRubrics (Baichuan-M2-32B). Time is average end-to-end consultation wall-clock time per case, including supervisor checks.

MaxturnPreceptorCoverageAvg. QA TurnsDoctor Tokens /CasePreceptor Tokens /CasePreceptor Calls/CaseRubrics / 1K Doctor TokensTime (s)
6--32.19%5.79184500.002.33917.7
6✓32.29%5.79186312781.002.32418.9
10--32.22%6.68277200.001.55920.5
10✓37.12%6.99300926971.911.65624.0
12--32.65%6.94294200.001.49321.1
12✓39.73%7.30328628502.011.62725.3

For Tables 4–5, "Rubrics / 1K Doctor Tokens" is the number of satisfied clinical rubrics per 1,000 Doctor LLM input/output tokens

Reviewer_gu2iOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

MedPreceptor is an adviser for language models conducting medical interviews with simulated patients. It follows the conversation, decides when guidance is needed, and suggests how the interviewing model could improve its next question. Both the supervisor and its larger teacher model are first fine-tuned on annotated consultations. The supervisor is then refined through on-policy distillation, with teacher feedback gated by whether its intervention decision is correct. When building the training data, the authors recheck proposed interventions using only what was known at that point in the conversation. The study asks whether this guidance helps different medical models gather more useful information rather than only focusing solely on the final diagnosis correctness.

Critical strengths

There’s a solid focus on usability. The model is compact and can guide arbitrary doctor models without base model fine-tuning. Improvements across multiple doctor models support its usefulness as a reusable supervisor.

I appreciate the prefix-only adjudication safeguard for handling hindsight bias. The comparison against standard OPD supports withholding positive teacher feedback when the intervention decision is wrong. The clinical combination and CACI are useful contributions, with clear precedents for individual components.

Critical weaknesses

I couldn’t find a comparison that keeps MedPreceptor’s intervention policy and reminder delivery unchanged, but replaces its tailored advice with a generic message such as ‘Review the conversation and reconsider your next question.’ This would help establish how much the gains depend on situation-specific advice, beyond prompting the doctor model to reconsider at the right moments.

The reported gains leave the effects of unnecessary interventions unclear, and with such poor precision, this seems important. Communicating how many incorrect interventions there are per consultation, or directly testing their effects, would make their practical impact clearer.

Other concerns: The reuse rationale is clear, but the tradeoffs against training a process-aware doctor deserve more discussion, and possibly even experimentation? Please report total consultation wall-clock time and token usage, including supervisor checks.

Questions for authors

What happens to diagnostic accuracy and information coverage if MedPreceptor uses the same intervention policy but delivers a fixed reconsideration reminder instead of tailored advice?

What happens to downstream performance when interventions judged unnecessary are suppressed (a counterfactual eval), while keeping the supervisor and advice-generation procedure otherwise unchanged?

5 comments

We thank the reviewer for the constructive feedback. We conducted new experiments on generic messages, effects of unnecessary interventions, and computational overhead to address each question below.

Q1: Does situation-specific advice provide gains beyond a generic reconsideration prompt?

Response to Q1: Situation-specific Advices vs. Generic Reminders

To evaluate the gain of situation-specific advices, we kept MedPreceptor’s intervention policy and reminder delivery unchanged, but replaces its tailored advice with a generic message such as "Review the conversation and reconsider your next question." We compare the performance of these two preceptor approaches with different Doctor models on the MAQuE dataset, as shown in Table 1

  1. Diagnostic accuracy: Situation-specific advice improves accuracy by 9.10, 4.62, and 0.69 percentage points, across Baichuan-M2-32B, GLM-5.3-Flash, and GPT-5.6-Sol, respectively, compared to generic reminders.
  2. Coverage and efficiency: Situation-specific advice increases clinical fact coverage on both Baichuan-M2-32B (+0.0329) and GLM-5.3-Flash (+0.0280) while requiring fewer dialogue turns across all three models. On GPT-5.6-Sol, the generic message achieves higher coverage, but situation-specific advice reaches higher diagnostic accuracy and relevance in nearly half the turns (8.601 vs. 4.919).

Table 1. Generic Reminder vs. Situation-Specific Advice on MAQuE.

DoctorAdvicesDoc. Acc.GPT Acc.Cov.Rel.Adh.Coh.Clar.Emp.Turns
Baichuan-M2-32BGeneric51.6266.410.32890.87100.85610.60900.68680.407314.947
Baichuan-M2-32BSituation-specific60.7268.240.36180.90080.87520.65030.68170.409414.720
GLM-5.3-FlashGeneric69.2170.790.36700.87330.98070.86630.81570.534612.965
GLM-5.3-FlashSituation-specific73.8373.000.39500.87300.98700.87600.79800.501012.088
GPT-5.6-SolGeneric73.9373.930.30550.89970.98710.87440.77800.40948.601
GPT-5.6-SolSituation-specific74.6274.620.23300.93700.96600.87400.61300.39404.919

Overall, these results indicate that tailored advice provides benefits beyond generic reminders, with trade-offs that vary across Doctor models.

Q2: What is the downstream impact of unnecessary interventions?

Responses to Q2: Effects of Unnecessary Interventions

To directly quantify the effects of unnecessary interventions, we performed a counterfactual rollout evaluation on CACI using clinical expert ground-truth labels to separate Necessary Interventions (TP) from Unnecessary Interventions (FP). From each consultation prefix, we continued the dialogue to completion both with and without a single intervention. Because expert necessity labels are available only for the current intervention, we disabled subsequent interventions in both branches to isolate its effect on final diagnostic accuracy and information coverage.

  1. In TP scenarios (Table 2) where interventions are necessary, providing intervention raises diagnostic accuracy by 2.07, 1.98, and 1.35 percentage points across the three advice sources; coverage also increases.
  2. In FP scenarios (Table 3) where interventions are unnecessary, providing intervention changes accuracy by +0.75, -0.24, and 0.00 percentage points, while coverage increases in all three groups. These results suggest that when interventions are clinically unnecessary, they still potentially help the Doctor gather useful clinical information, although their effects on diagnostic accuracy vary.

Table 2. Effects of Necessary Intervention on True-Positive (TP) Scenarios in CACI.

Advice SourceSettingNGPT Acc.Cov.Rel.Adh.Coh.Clar.Emp.
MedPreceptorWithout Advice59146.990.31550.94470.91240.72780.73720.4748
MedPreceptorWith Advice59149.060.33940.95540.91650.73050.73230.4703
GPT-5.6-SolWithout Advice42247.030.31290.94550.90300.70540.73270.4703
GPT-5.6-SolWith Advice42249.010.32360.95260.89550.70150.72430.4639
DeepSeek-V4.1-FlashWithout Advice38647.700.30830.94710.89700.69160.73330.4683
DeepSeek-V4.1-FlashWith Advice38649.050.32320.95190.89650.69430.72090.4721

Table 3. Effects of Unnecessary Interventions on False-Positive (FP) Scenarios in CACI.

Advice SourceSettingNGPT Acc.Cov.Rel.Adh.Coh.Clar.Emp.
MedPreceptorWithout Advice43540.900.29950.94810.95610.81430.76100.4874
MedPreceptorWith Advice43541.650.33250.94570.93550.79960.76170.4893
GPT-5.6-SolWithout Advice32143.130.29280.95690.94890.78660.75340.4792
GPT-5.6-SolWith Advice32142.890.31750.96410.93930.77890.73990.4837
DeepSeek-V4.1-FlashWithout Advice28441.160.28880.95280.94510.77910.74870.4838
DeepSeek-V4.1-FlashWith Advice28441.160.31840.95100.94220.78340.75020.4874

In both TP and FP settings, secondary dialogue metrics (adherence, coherence, clarity, and empathy) fluctuate by less than 0.021. Because MedPreceptor’s advice is grounded strictly in visible prefix facts, unnecessary interventions act as benign prompts to inquire additional history rather than derailing diagnostic reasoning.

Q3: What are the trade-offs of a decoupled supervisor versus a process-aware Doctor, and what are the time and token costs?

Responses to Q3: Decoupling Design Trade-offs and Efficiency

Trade-offs against a process-aware Doctor. While training an end-to-end process-aware Doctor is a viable alternative, coupling process regulation with consultation execution introduces three key limitations:

  1. Capability interference: Active inquiry and diagnosis capabilities often compete within a single model (Li et al., 2024; Fang et al., 2026; Chen et al, 2026).
  2. Retraining overhead: End-to-end multi-turn RL requires costly rollouts for every model update and cannot be applied to proprietary APIs, whereas our 4B supervisor trains once and transfers zero-shot.
  3. Complementary gains and human support: An external process supervisor can also support human clinicians in existing clinical workflows.

We also evaluated end-to-end consultation latency and token usage (including all supervisor checks) on MedDialogRubrics under both a fixed maximum-turn budget (Table 4) and a strict 6K total-token cap per case (Table 5). We report Doctor and Preceptor token usage separately because their per-token costs differ substantially. Rubrics per 1K Doctor tokens measures the efficiency of using the more expensive Doctor model, rather than total system cost. End-to-end wall-clock time captures the deployment latency, including all supervisor checks.

  1. In Table 4, MedPreceptor is invoked only 1.00, 1.91, and 2.01 times per case across the three turn budgets, raising efficiency from 1.559 to 1.656 rubrics per 1K Doctor tokens at 10 turns (and from 1.493 to 1.627 at 12 turns) with minimal latency overhead.
  2. Across the first 1,600 MedDialogRubrics cases under a shared (Doctor+Preceptor) token ceiling (Table 5), MedPreceptor averages 0.79 calls per case and increases per-turn efficiency from 2.033 to 2.175 rubrics per turn, confirming that external supervision improves information gathering even under the same total-token cap.

We will include these results and the trade-off discussion in the revised manuscript.

Table 4. Fixed Maximum-Turn Budget on MedDialogRubrics (Baichuan-M2-32B). Time is average end-to-end consultation wall-clock time per case.

MaxturnPreceptorCoverageAvg. QA TurnsDoctor Tokens / CasePreceptor Tokens / CasePreceptor Calls / CaseRubrics / 1K Doctor TokensTime (s)
6--32.19%5.79184500.002.33917.7
6Yes32.29%5.79186312781.002.32418.9
10--32.22%6.68277200.001.55920.5
10Yes37.12%6.99300926971.911.65624.0
12--32.65%6.94294200.001.49321.1
12Yes39.73%7.30328628502.011.62725.3

Table 5. Fixed Total-Token Budget on MedDialogRubrics (GPT-5.6-Sol, 1,600 cases).

PreceptorCoverageAvg. QA TurnsDoctor Tokens / CasePreceptor Tokens / CasePreceptor Calls / CaseRubrics / TurnRubrics / 1K Doctor Tokens
--54.48%3.51843006.770.000.00002.0332.379
Yes56.37%3.40842745.591315.980.79492.1752.700
Reviewer_N4EAOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a form of test-time supervision. Given a student's trajectory HtH_t, a teacher/supervisor can electively (dt=1d_t = 1) interject with corrective instruction ata_t and an explanation ete_t for this intervention.

Its main contribution is a benchmark, called CACI, in which three graduate students labeled 1664 trajectories in this manner. Supervisor models are evaluated on both dtd_t and at∣dt=1a_t | d_t = 1.

To train a small supervisor model (MedPreceptor), a frontier model (GPT 5.5) generates such data (Ht,dt∗,et∗,at∗)(H_t, d_t^*, e_t^*, a_t^*). Then, both 4B and 27B supervisors are SFT'd on this data. These are the student and teacher, respectively, in subsequent on-policy distillation. However, this OPD step has a modification: if the student d^t\hat{d}_t disagreed with GPT dt∗d^*_t, then only "negative" teacher supervision (where the teacher assigned lower probability than the student) is kept. The motivation: if the student didn’t make the right intervention decision, its explanation/rationale shouldn't receive higher probability. This modification is called "DG-OPD".

The main experiments compare (1) LLM alone versus (2) LLM paired with 4B supervisor. There is an additional experiment where the effect of base -> SFT -> OPD supervisor is examined. Experimental gains are mostly attributed to the supervision setup and SFT.

Critical strengths
  • Attaching a 4B model as a supervisor is a cheap way to improve performance.

  • Good mix of models used in experiments.

Critical weaknesses
  • Unclear motivation. The overall goal of this paper is to train a very small (4B) supervisor to approximate a 27B one. However, a supervisor does not need to perform generation; in practice, it can actually be substantially larger than the LLM being supervised. Efficiency is always good, but it is unclear whether this is the most impactful place to seek it.

  • No comparison to other supervisors. This supervision architecture is not novel. (See the uncited work of [1], for example). This paper does not compare to other supervision methods that address the same problem,

  • DG-OPD is not very novel, conceptually questionable, and of limited value. DG-OPD is just a minor tweak of OPD. Furthermore, this tweak does not seem obviously well-designed: if the student made the wrong decision to intervene, then why supervise its subsequent advice text at all, at a token level? DG-OPD does not lead to much improvement beyond SFT: out of the 10 points of accuracy improvement, DG-OPD adds just 3.2; the rest are due to the presence of an SFT'd supervisor.

  • CACI has limited residual value as an eval. The labeling setup (3 graduate-level humans) is not rigorous for CACI to serve as a standalone evaluation.

[1] Lin, Fulin, et al. "Stop wasting your tokens: Towards efficient runtime multi-agent systems." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors

Why was the Sequential Diagnosis Benchmark of Nori et al (2026) not tried? While this benchmark was published only recently, it has been on arXiv/GitHub since June 2025.

[1] Nori, Harsha, et al. "Sequential diagnosis with language models." Science 394.6822 (2026): 423-428.

12 comments

We thank the reviewer for the constructive feedback. We have added a comparison with an existing supervisor on SDBench, further DG-OPD ablations, and annotation agreement results. We respond to each concern below.

Q1: Why train a compact 4B supervisor instead of using a larger model?

Response to Q1: Motivation for a Compact Supervisor

MedPreceptor performs both intervention detection and context-specific advice generation; it is not solely an intervention classifier. Our goal is to develop a compact, reusable supervisor that helps both Doctor LLMs and human clinicians gather clinical information more effectively, make better diagnostic decisions, and reduce the risk of missed diagnoses. Rather than simply training a 4B model to approximate a 27B supervisor, we aim to provide timely guidance during consultations without modifying the Doctor LLM's weights, while keeping the model small enough for practical deployment in real clinical settings.

1. Local deployment in hospitals. Many hospital consultation and electronic medical record (EMR) systems run on restricted internal networks to protect patient data, where external cloud APIs may not be available. Local computing resources can be limited, and hardware upgrades are costly. A 4B supervisor is more practical for local deployment than a much larger model, especially when it needs to check the consultation repeatedly.

2. Efficiency and clinical performance. We chose 4B to balance response speed, memory use, and supervision quality. In our additional experiment with Baichuan-M2-32B at Maxturn = 12, MedPreceptor improves information coverage from 32.65% to 39.73%, while end-to-end consultation time increases from 21.1 to 25.3 seconds per case, including supervisor checks.

3. Long-term goal. Beyond supporting different Doctor LLMs, we hope to extend MedPreceptor to real clinical settings, where it could help human clinicians identify missing information and conduct more effective consultations. A compact model that can run locally would make this direction more practical.


Q2: How does MedPreceptor compare with existing supervision methods?

Response to Q2: Comparison with Existing Supervisors

We added a comparison with SupervisorAgent (Lin et al., ICLR 2026) on SDBench. Both methods determine whether intervention is needed at each consultation turn. But SupervisorAgent relies on general-purpose heuristic rules to trigger its supervisor LLM (DeepSeek-V4.1-Flash). These rules may overlook clinical inquiry issues, such as missing key information or misdirected questioning, whereas MedPreceptor learns intervention decisions directly from clinical consultation data.

As shown in Table 1, MedPreceptor achieves 43.03% diagnostic accuracy on Baichuan-M2-32B, compared with 36.65% without supervision and 37.45% with SupervisorAgent. On GLM-5.3-Flash, the corresponding results are 60.87%, 55.59%, and 53.73%. MedPreceptor also reduces clinical examination costs relative to the no-helper baseline for both Doctors. These results highlight the value of clinical inquiry-specific supervision over general-purpose intervention triggers in this setting.

Table 1. Comparison with SupervisorAgent on SDBench. The best accuracy and lowest clinical cost within each Doctor are highlighted in bold. Supervisor LLM Calls/Case counts actual model invocations after heuristic filtering.

DoctorMethodAccuracyAvg. Info TurnsAvg. Clinical Exam TurnsSupervisor LLM Calls/CaseActual Interventions/CaseClinical Cost (USD/Case)
Baichuan-M2-32BNo Helper36.65%3.682.100.000.001339.62
Baichuan-M2-32BMedPreceptor43.03%4.171.904.172.141242.89
Baichuan-M2-32BSupervisorAgent37.45%3.672.050.080.061308.36
GLM-5.3-FlashNo Helper55.59%8.074.400.000.003517.37
GLM-5.3-FlashMedPreceptor60.87%8.464.168.364.853215.39
GLM-5.3-FlashSupervisorAgent53.73%7.694.090.170.133127.08

Q3: Why is DG-OPD designed this way, and how much does it improve over SFT?

Response to Q3: DG-OPD Design and Effectiveness

We appreciate the reviewer's questions about the design and additional value of DG-OPD. We address them through the training objective and new ablation results.

1. Why retain negative supervision? In MedPreceptor, the decision to intervene and the advice text are linked. If the student makes the wrong decision, DG-OPD does not treat the resulting advice as a positive target. Instead, it retains negative teacher signals to discourage the incorrect branch. We added an ablation that skips training updates for incorrect decisions. Its MAQuE Doctor accuracy is 53.90%, versus 60.72% for DG-OPD. This supports retaining negative supervision rather than dropping these branches entirely.

2. What does DG-OPD add beyond SFT? On MAQuE, DG-OPD improves Doctor accuracy by 3.20 points over SFT and 4.75 points over standard OPD. On CACI, it improves intervention-decision accuracy by 7.34 points and F1 by 7.04 points over SFT. These results show benefits for both intervention decisions and downstream consultations.

3. What is the broader contribution? DG-OPD is one part of our approach. More broadly, MedPreceptor introduces external process supervision to help Doctor LLMs gather clinical information without changing their weights. We also use two-stage annotation to reassess candidate interventions using only the visible dialogue prefix. In another ablation, we compared our two-stage annotation approach with Stage A-only SFT, where an LLM reviews the complete consultation trajectory to identify intervention points and generate advice. Two-stage SFT achieves 57.52% Doctor accuracy, compared with 53.86% for Stage A-only SFT, supporting the benefit of reassessing interventions using only the current dialogue prefix to avoid future information leakage during annotation.

Table 2. Ablation Study of MedPreceptor Training Strategies on MAQuE.

MethodDoctor Acc.GPT Acc.Cov.Rel.Adh.Coh.Cla.Emp.Q.
No Preceptor50.4566.340.31300.87900.84600.57600.68200.404014.85
Base52.5564.070.33130.89830.85880.61190.67660.409315.02
SFT57.5266.830.35460.89480.86870.63710.68210.409714.71
Stage A-only SFT53.8666.340.35200.88040.88550.66130.68370.409514.46
OPD55.9765.930.35020.89280.85980.62790.68630.408714.85
OPD (Skip Wrong Decisions)53.9066.000.35040.88110.87340.64080.68460.408114.79
DG-OPD60.7268.240.36180.90080.87520.65030.68170.409414.72
Teacher60.3466.550.34860.89180.87220.63850.68370.410014.64

Note: Stage A-only uses single-stage annotation. OPD (Skip Wrong Decisions) skips updates when the student makes an incorrect intervention decision.

Q4: How reliable are the CACI annotations, and what is the benchmark's value?

Response to Q4: Reliability and Value of CACI

We thank the reviewer for highlighting the importance of clinical expertise and reliable annotations. CACI contains 1,664 expert-annotated consultation states. The three annotators each hold a Master's degree in Clinical Medicine, have completed standardized residency training, hold residency certificates, and have over five years of cumulative clinical experience.

1. Annotation reliability. We added agreement results before reconciliation. The three pairwise intervention-decision agreement rates are 85.04%, 90.87%, and 89.96%, with Cohen's κ ranging from 0.679 to 0.813 (Table 3). Across the CACI records, 82.93% of intervention decisions were unanimous (multi-rater Gwet's AC1 = 0.782). The remaining 17.07% had a 2:1 split, which was resolved by majority vote.

When the majority voted to intervene, we compared the advice-topic label sets of all experts supporting intervention. If all label sets matched, we randomly selected one of their suggestions. Otherwise, the experts discussed the differences to form the final advice. Under this rule, only 7.87% of the CACI records required advice reconciliation.

We also report agreement on advice topics (Table 4), using labels consistently mapped from the experts' suggestion texts. Overall, the results show good agreement across most advice topics, supporting the consistency of the experts' advice content.

2. Evaluation value. CACI measures whether an intervention is needed and whether its advice is appropriate, rather than relying only on the final diagnosis. This matters because a Doctor can miss important symptoms, history, or medication information even when its final diagnosis is plausible. CACI therefore complements outcome-based benchmarks by evaluating the information-gathering process.

Table 3. Pairwise Inter-Expert Agreement on Intervention Decisions.

ComparisonAgreement (%)Disagreement (%)Cohen's κGwet's AC1
Expert 1–Expert 285.0414.960.6790.719
Expert 1–Expert 390.879.130.8130.822
Expert 2–Expert 389.9610.040.7940.806

Table 4. Per-Topic Inter-Expert Agreement on Advice Content (Gwet's AC1).

Advice TopicExpert 1–Expert 2Expert 1–Expert 3Expert 2–Expert 3Three-Expert AC1
Medication dose / timing0.9540.9720.9790.968
Repeated questioning0.7580.8530.8760.830
Abdominal pain / vomiting / bowel0.9900.9901.0000.993
Triggers and symptoms0.9570.9760.9750.969
Nonexclusive diagnosis1.000*1.000*1.000*1.000*
Relevant medical history0.9360.9640.9670.956
Other0.8690.9490.8990.906

Note: The asterisk (*) indicates that no positive instances were found for the nonexclusive-diagnosis topic; its agreement value reflects unanimous absence.

Agreement by clinical specialty. We also examined intervention-decision agreement before reconciliation across six specialties (Table 5). The three-expert unanimous agreement ranges from 76.03% to 90.22%, with an overall rate of 82.93%. The corresponding multi-rater Gwet's AC1 ranges from 0.688 to 0.888.

Table 5. Pre-Reconciliation Agreement on Intervention Decisions by Clinical Specialty. Each cell reports agreement rate / Gwet's AC1. The "Three-expert" column reports unanimous agreement and multi-rater AC1; the other columns report pairwise agreement between human experts.

Clinical SpecialtyThree-expert Agreement / AC1Expert 1–Expert 2Expert 1–Expert 3Expert 2–Expert 3
Respiratory Medicine83.33% / 0.78083.33% / 0.67683.33% / 0.676100.00% / 1.000
Cardiology76.03% / 0.68880.58% / 0.62788.43% / 0.77283.06% / 0.669
General Internal Medicine80.51% / 0.74383.90% / 0.68689.41% / 0.78887.71% / 0.758
Gastroenterology90.22% / 0.88891.67% / 0.86095.65% / 0.92393.12% / 0.881
Hematology80.24% / 0.73784.98% / 0.70090.12% / 0.80485.38% / 0.708
Rheumatology and Immunology85.01% / 0.82585.01% / 0.74695.61% / 0.92089.41% / 0.815
Overall82.93% / 0.78285.04% / 0.71990.87% / 0.82289.96% / 0.806

Comments on the paper

1 comment

We sincerely thank all reviewers for their careful evaluations, constructive feedback, and thoughtful discussions. Our work aims to improve clinical information gathering in multi-turn medical consultations through selective guidance. MedPreceptor provides lightweight, external supervision across different Doctor LLMs and has the potential to support human clinicians. During the rebuttal, we conducted additional experiments and clinical evaluations to address the reviewers' concerns and further strengthen our work.

We appreciate the reviewers' recognition of our key contributions:

  • Practical and Model-Agnostic Supervision: Reviewers gu2i, hody, and wEVU highlighted MedPreceptor's lightweight and reusable design, which improves clinical inquiry without modifying Doctor LLM parameters.
  • Methodological Design: Reviewers gu2i and wEVU appreciated our two-stage prefix-aligned annotation strategy, while gu2i and hody recognized the value of DG-OPD in addressing limitations of standard OPD.
  • Clinical Process Evaluation: Reviewers gu2i and wEVU recognized the value of CACI. Reviewer N4EA further acknowledged its annotation reliability and the practical value of supporting human clinicians.

Below, we summarize our major additional experiments and how they address the reviewers' concerns.

  • Clinical Expert Validation and CACI Reliability (Reviewers wEVU, N4EA, and hody):

    • Blinded Clinical Evaluation: We evaluated 384 anonymized suggestions across 64 consultation prefixes. MedPreceptor achieved expert scores of 11.17 and 11.72 (out of 12), comparable to GPT-5.6-Sol (11.14 and 11.80), supporting its clinical advice quality.
    • Annotation Agreement: Three clinical experts achieved 82.93% unanimous agreement (Gwet's AC1 = 0.782) on CACI, with only 7.87% of cases requiring advice reconciliation.
  • Selective Intervention and Tailored Advice (Reviewers gu2i and hody):

    • Generic Reminder: With the same intervention policy, tailored advice improved Baichuan-M2-32B diagnostic accuracy from 51.62% to 60.72%, showing benefits beyond additional prompting.
    • Fixed-Frequency Intervention: MedPreceptor achieved 60.72% accuracy, outperforming fixed interventions every 3 turns (54.48%) and 6 turns (52.69%), confirming the value of selective intervention.
    • Unnecessary Interventions: Counterfactual experiments showed increased information coverage with limited effects on diagnostic accuracy.
  • DG-OPD and Training Ablations (Reviewers N4EA, gu2i, and wEVU):

    • DG-OPD Ablations: DG-OPD achieved 60.72% diagnostic accuracy, outperforming SFT (57.52%), standard OPD (55.97%), and OPD Masking (55.93%), supporting decision-guided supervision.
    • Two-Stage Annotation: Prefix-aligned annotation improved SFT accuracy from 53.86% to 57.52%, supporting its effectiveness in avoiding future information leakage. We also clarified the rationale behind DG-OPD's decision-conditioned teacher signals.
  • Cross-Model Generalization and Efficiency (Reviewers hody, wEVU, and gu2i):

    • Generalization: Three unseen Doctor models achieved +2.65 to +5.24 percentage points in diagnostic accuracy, with paired bootstrap 95% confidence intervals excluding zero for all five evaluated models.
    • Budget-Matched Evaluation: Under a shared 6K-token budget, coverage improved from 54.48% to 56.37% on GPT-5.6-Sol. With a 12-turn limit, coverage increased from 32.65% to 39.73% on Baichuan-M2-32B. We also reported token usage and end-to-end latency to clarify computational trade-offs.
  • External Benchmark and Existing Methods (Reviewer N4EA):

    • SDBench Evaluation: MedPreceptor improved diagnostic accuracy from 36.65% to 43.03% on Baichuan-M2-32B and 55.59% to 60.87% on GLM-5.3-Flash, outperforming SupervisorAgent.
    • Inference-Time Comparison: MedPreceptor achieved accuracy close to MAI-DxO-style Five-Doctor Orchestration, which required 11.5–12.9× more model tokens and incurred higher estimated clinical examination costs, highlighting our framework's practical advantages.

Conclusion

Our additional experiments and clinical evaluations have further strengthened our experimental evidence.

We believe that MedPreceptor offers more than improvements in medical LLM performance. Its lightweight, reusable supervision framework improves clinical information gathering without modifying existing Doctor models. We also believe it has the potential to be deployed in real hospitals, integrating doctor–patient conversations and electronic health records to provide timely, non-intrusive guidance to human clinicians.

We sincerely thank all reviewers again for their valuable feedback and continued engagement, which have helped us strengthen our work and better demonstrate its practical value.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.