- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes an independent supervision model that monitors doctor–patient conversations and offers guidance when the doctor model repeats questions, misses clues, or reaches conclusions prematurely. Contributions include a plug-in selective supervision framework, a two-stage annotation process that avoids using future information in final supervision, and Decision-Guided Selective On-Policy Distillation (DG-OPD). The key design is to retain normal teacher guidance when the student’s intervention decision is correct, but retain only suppressive signals when it is incorrect, avoiding reinforcement of an erroneous decision. The authors generate consultation trajectories from multiple doctor models, use complete conversations to identify candidate intervention points, and then reassess those points using only the information available at that moment. Teacher and student supervisors are trained to output an intervention decision, supporting evidence, and corrective advice. During distillation, the learning signal depends on whether the intervention decision is correct. At deployment, the supervisor reads the ongoing conversation and provides next-step guidance when needed.
Critical strengths
- This paper offers a practically useful modular design. The supervisor supports multiple open- and closed-source doctor models without modifying their parameters, with relatively consistent improvements across models.
- Aligns methodological choices with clearly defined challenges. Two-stage annotation restricts future information from entering supervision, DG-OPD adjusts learning signals based on intervention correctness, and CACI separately evaluates intervention capability.
Critical weaknesses
- The clinical reliability of generated advice lacks independent expert validation. Human annotations in CACI support intervention-decision evaluation, but advice quality is primarily judged by GPT-5.6-Sol, without reported agreement with clinical experts. Higher automated scores therefore do not establish that the advice is safe, appropriate, or beneficial to subsequent consultation. Authors should conduct blinded clinical expert evaluation of representative outputs, including unnecessary interventions, assessing correctness, necessity, actionability, and potential harm. Report inter-expert agreement and agreement between experts and the LLM judge.
- Efficiency gains are inconsistent, and budget-matched comparisons are missing. For GPT-5.6-Sol, average consultation length increases from 5.69 to 7.38 turns, while satisfied rubrics per turn decrease from 1.234 to 1.029, suggesting that higher coverage may partly result from longer conversations. Although other models show improved per-turn efficiency, the additional system-level overhead remains unclear. Compare performance under fixed dialogue-turn and total-token budgets, including supervisor calls, and report coverage, latency, monetary cost, and information gained per unit cost.
Questions for authors
- Appendix B.3 instructs annotators not to label alternative but clinically reasonable inquiry strategies as errors. What was the agreement among the three CACI annotators before adjudication? Could the authors provide agreement statistics and representative borderline cases showing how this principle was applied consistently?
- Although Stage B uses only the current dialogue prefix, candidate intervention points are selected by Stage A with access to the complete conversation and final diagnosis. Have the authors assessed whether this selection process misses problems identifiable from the prefix alone or disproportionately selects particular error types?
6 comments
Q2: Does MedPreceptor remain effective under matched turn and token budgets, and what is its system-level overhead?
Response to Q2: Budget-Matched Efficiency and System Overhead
We conducted fixed total-token and fixed maximum-turn comparisons on MedDialogRubrics. We report coverage, consultation length, decomposed Doctor and Preceptor token usage, supervisor call frequency, end-to-end latency, and information gained per Doctor token. We report Rubrics / 1K Doctor Tokens because MedPreceptor can be deployed locally at relatively low cost, whereas Doctor LLMs are typically much more expensive to run. This metric therefore emphasizes information-gathering efficiency relative to the dominant inference cost.
1. Fixed Total-Token Budget. We evaluated GPT-5.6-Sol across 1,600 MedDialogRubrics cases under an externally enforced 6K total-token cap per case that counts both Doctor and Preceptor input/output tokens together. As shown in Table 3, MedPreceptor increases coverage from 54.48% to 56.37%, while reducing average consultation length from 3.5184 to 3.4084 turns. Satisfied rubrics per turn also increase from 2.033 to 2.175.
Table 3. Fixed 6K Total-Token Budget on MedDialogRubrics (GPT-5.6-Sol, 1,600 cases).
| Preceptor | Macro Coverage | Avg. QA Turns | Doctor Tokens /Case | Preceptor Tokens /Case | Preceptor Calls/Case | Rubrics /Turn | Rubrics / 1K Doctor Tokens |
|---|---|---|---|---|---|---|---|
| No | 54.48% | 3.5184 | 3006.77 | 0.00 | 0.0000 | 2.033 | 2.379 |
| Yes | 56.37% | 3.4084 | 2745.59 | 1315.98 | 0.7949 | 2.175 | 2.700 |
2. Fixed Maximum-Turn Budget. We further compared GPT-5.6-Sol (2,000 cases, Table 4) and Baichuan-M2-32B (Table 5) under matched maximum-turn caps. Across the three turn limits (Maxturn = 6,10,12), MedPreceptor improves rubric coverage by 5.71, 3.52, and 3.27 percentage points on GPT-5.6-Sol, and by 0.10%, 4.90%, and 7.08% on Baichuan-M2-32B. Average Preceptor calls per case across the three turn caps in Table 4 are 0.8430, 1.0745, and 1.6720.
Table 4. Fixed Maximum-Turn Budget on MedDialogRubrics (GPT-5.6-Sol, 2,000 cases).
| Maxturn | Preceptor | Coverage | Avg. QA Turns | Doctor Tokens /Case | Preceptor Tokens /Case | Preceptor Calls/Case | Rubrics / 1K Doctor Tokens |
|---|---|---|---|---|---|---|---|
| 6 | -- | 49.83% | 3.9100 | 3444.08 | 0.00 | 0.0000 | 1.921 |
| 6 | ✓ | 55.54% | 4.0670 | 3570.49 | 1387.81 | 0.8430 | 2.065 |
| 10 | -- | 53.50% | 4.0095 | 3950.23 | 0.00 | 0.0000 | 1.798 |
| 10 | ✓ | 57.02% | 4.3190 | 4361.62 | 1917.43 | 1.0745 | 1.736 |
| 12 | -- | 53.81% | 5.2235 | 4497.25 | 0.00 | 0.0000 | 1.588 |
| 12 | ✓ | 57.08% | 6.4365 | 6509.24 | 3004.60 | 1.6720 | 1.164 |
3. Latency and cost. To avoid network variance from external commercial APIs, we measured end-to-end consultation wall-clock time on locally deployed Baichuan-M2-32B, including all supervisor checks (Table 5). At Maxturn = 6, 10, and 12, MedPreceptor adds 1.2, 3.5, and 4.2 seconds of end-to-end consultation time per case, respectively, while clinical coverage substantially expands. We also report Rubrics / 1K Doctor Tokens, the number of satisfied clinical rubrics per 1,000 Doctor input/output tokens. This measures efficiency relative to Doctor token usage. A single monetary-cost figure depends on the API pricing or local hardware used, so we report the measured token and time costs instead of assuming a universal dollar cost.
Table 5. Fixed Maximum-Turn Budget and Latency on MedDialogRubrics (Baichuan-M2-32B).
| Maxturn | Preceptor | Coverage | Avg. QA Turns | Doctor Tokens /Case | Preceptor Tokens /Case | Preceptor Calls/Case | Rubrics / 1K Doctor Tokens | Time (s) |
|---|---|---|---|---|---|---|---|---|
| 6 | -- | 32.19% | 5.79 | 1845 | 0 | 0.00 | 2.339 | 17.7 |
| 6 | ✓ | 32.29% | 5.79 | 1863 | 1278 | 1.00 | 2.324 | 18.9 |
| 10 | -- | 32.22% | 6.68 | 2772 | 0 | 0.00 | 1.559 | 20.5 |
| 10 | ✓ | 37.12% | 6.99 | 3009 | 2697 | 1.91 | 1.656 | 24.0 |
| 12 | -- | 32.65% | 6.94 | 2942 | 0 | 0.00 | 1.493 | 21.1 |
| 12 | ✓ | 39.73% | 7.30 | 3286 | 2850 | 2.01 | 1.627 | 25.3 |
Q3: What is the pre-reconciliation inter-annotator agreement in CACI, and how are clinically reasonable borderline cases handled?
Response to Q3: CACI Annotation Agreement and Borderline Cases
We report an additional reconciliation review by three human experts.
1. Pre-reconciliation decision agreement. Before adjudication, all three clinical experts unanimously agreed on 82.93% of the CACI records (multi-rater Gwet's AC1 = 0.782), while remaining 17.07% exhibited a 2:1 split resolved by majority vote. Pairwise agreement across the benchmark is summarized in Table 6.
Table 6. Pairwise Inter-Expert Agreement on CACI Intervention Decisions.
| Comparison | Agreement (%) | Disagreement (%) | Cohen's κ | Gwet's AC1 |
|---|---|---|---|---|
| Expert 1--2 | 85.04 | 14.96 | 0.679 | 0.719 |
| Expert 1--3 | 90.87 | 9.13 | 0.813 | 0.822 |
| Expert 2--3 | 89.96 | 10.04 | 0.794 | 0.806 |
2. Advice reconciliation and topic consistency. When the majority voted to intervene, we compared the advice-topic label sets from all experts supporting intervention. If all sets matched, one supporting suggestion was selected at random; if any set differed, the experts discussed the case to reach a final suggestion. Under this protocol, only 7.87% of CACI records required advice reconciliation, with high inter-expert agreement across individual clinical advice topics (Table 7).
Table 7. Per-Topic Inter-Expert Agreement on Advice Content (Gwet's AC1).
| Advice Topic | Expert 1–Expert 2 | Expert 1–Expert 3 | Expert 2–Expert 3 | Three-Expert AC1 |
|---|---|---|---|---|
| Medication dose / timing | 0.954 | 0.972 | 0.979 | 0.968 |
| Repeated questioning | 0.758 | 0.853 | 0.876 | 0.830 |
| Abdominal pain / vomiting / bowel | 0.990 | 0.990 | 1.000 | 0.993 |
| Triggers and symptoms | 0.957 | 0.976 | 0.975 | 0.969 |
| Nonexclusive diagnosis | 1.000* | 1.000* | 1.000* | 1.000* |
| Relevant medical history | 0.936 | 0.964 | 0.967 | 0.956 |
| Other | 0.869 | 0.949 | 0.899 | 0.906 |
* No positive instances occurred for the nonexclusive-diagnosis topic; the value 1.000 reflects unanimous absence, not agreement on detecting that topic.
3. Agreement across clinical specialties. Pre-reconciliation decision agreement also remains consistently strong across all six clinical specialties in CACI (Table 8). The three-expert unanimous agreement ranges from 76.03% to 90.22%, with an overall rate of 82.93%. Multi-rater Gwet's AC1 ranges from 0.688 to 0.888 across specialties.
Table 8. Pre-Reconciliation Decision Agreement by Clinical Specialty.
| Clinical Specialty | Three-Expert | Expert 1--2 | Expert 1--3 | Expert 2--3 |
|---|---|---|---|---|
| Respiratory Medicine | 83.33% / 0.780 | 83.33% / 0.676 | 83.33% / 0.676 | 100.00% / 1.000 |
| Cardiology | 76.03% / 0.688 | 80.58% / 0.627 | 88.43% / 0.772 | 83.06% / 0.669 |
| General Internal Medicine | 80.51% / 0.743 | 83.90% / 0.686 | 89.41% / 0.788 | 87.71% / 0.758 |
| Gastroenterology | 90.22% / 0.888 | 91.67% / 0.860 | 95.65% / 0.923 | 93.12% / 0.881 |
| Hematology | 80.24% / 0.737 | 84.98% / 0.700 | 90.12% / 0.804 | 85.38% / 0.708 |
| Rheumatology and Immunology | 85.01% / 0.825 | 85.01% / 0.746 | 95.61% / 0.920 | 89.41% / 0.815 |
| Overall | 82.93% / 0.782 | 85.04% / 0.719 | 90.87% / 0.822 | 89.96% / 0.806 |
Each cell in Table 8 reports raw agreement rate followed by Gwet's AC1 (Three-Expert denotes unanimous agreement and multi-rater AC1; remaining columns report pairwise agreement).
4. Representative borderline cases. The following two 2:1 split cases illustrate how our protocol preserves alternative yet clinically reasonable inquiry strategies without misclassifying them as errors:
-
Case 1 (Cardiovascular----symptom refinement vs. repetition): An 85-year-old man with coronary artery disease and a prior bare-metal stent presented with worsening exertional chest pain and dyspnea. After asking about shortness of breath at Turn 7 (eliciting exertional dyspnea), the Doctor asked about chest tightness and breathlessness at Turn 10; while Expert 2 flagged this as repetitive (INTERVENE), Experts 1 and 3 judged it a clinically valid differentiation of cardiopulmonary symptoms, yielding a majority decision of NO_INTERVENE.
-
Case 2 (Hematology----competing clinical priorities): A 62-year-old man with celiac disease and prior gastrointestinal anastomoses presented with severe anemia (hematocrit 20), a positive fecal occult blood test, and unspecified discomfort following accidental gluten exposure. At Turn 4, the Doctor prioritized questions about possible gastrointestinal bleeding, including melena and hematochezia. Although Expert 1 preferred prioritizing the gluten trigger (INTERVENE), Experts 2 and 3 judged ruling out acute blood loss an appropriate clinical priority, resulting in a majority decision NO_INTERVENE.
We thank the reviewer for the insightful review and constructive feedback. We conducted additional experiments and analyses to address the following questions:
Q1: How reliable is the generated advice when evaluated by blinded clinical experts, and how well does the LLM judge align with human raters?
Response to Q1: Blinded Clinical Expert Evaluation of Advice
1. Evaluation setup. We conducted a blinded evaluation across 64 representative consultation prefixes, including both necessary and unnecessary intervention states, where six models each generated guidance, yielding 384 anonymized suggestions. Two clinical experts and an LLM judge (GPT-5.6-Sol) independently scored each suggestion solely from the visible case context and dialogue prefix (without access to reference advice or model identity) across six dimensions from 0 to 2, for a maximum total score of 12. The score dimensions are Problem Alignment, Clinical Actionability, Grounding, Efficiency and Non-redundancy, Advisor Style and Safety, and Intervention Necessity.
2. Inter-expert and LLM-judge agreement. Table 1 reports unweighted (Gwet's AC1) and ordinal-weighted (Gwet's AC2) agreement, which scale from -1 (systematic disagreement) through 0 (chance) to +1 (perfect agreement). The two clinical experts show high consistency across all dimensions, particularly on Clinical Actionability (0.958), Grounding (0.953), and Intervention Necessity (0.959). The LLM judge aligns closely with both clinicians on core quality dimensions (Problem Alignment, Clinical Actionability, Grounding, and Advisor Style / Safety), validating its reliability for large-scale evaluation. Agreement on Efficiency / Non-redundancy is lower both between the LLM and experts and between the two clinicians themselves, possibly because different experts may adopt different yet reasonable inquiry strategies for the same case.
Table 1. Pairwise Agreement on Blinded Advice Ratings (Gwet's AC1 / AC2).
3. Advice quality and calibrated safety assessment. As shown in Table 2, MedPreceptor receives average scores of 11.17 and 11.72 from the two clinical experts, compared with 11.14 and 11.80 for GPT-5.6-Sol. Neither clinical expert nor the LLM judge flagged any suggestion in these samples as introducing direct clinical harm or unsafe patient guidance (0/384). While this indicates that prefix-grounded process reminders rarely introduce overt safety violations in the evaluated states, it does not guarantee error-free deployment: as analyzed in Appendix B.4, suboptimal question prioritization or unsupported diagnostic assumptions can still occur, and we have revised our manuscript claims accordingly.
Table 2. Blinded Advice-Quality Scores Across Evaluators.