- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents MeditronFO, a clinical-LLM development pipeline combining public medical QA, synthetic QA, guideline-derived questions, and synthetic clinical vignettes. The final corpus contains 533,888 examples, with answer distillation, label-based rejection sampling where applicable, decontamination, and audit procedures. The pipeline is applied to five fully-open bases and a Gemma-3-27B open-weight control. Apertus-70B improves from 48.35% to 54.56% on the reported five-benchmark average. The paper also introduces AutoMOOVE: judges are validated against 576 historical answer pairs with unanimous clinician preferences, and new models are compared on the corresponding 516 vignettes. Analyses examine corpus components, teacher choice, judge robustness, response length, and general-capability degradation.
Critical strengths
S1. A useful resource-oriented development contribution.
The manuscript specifies a reproducible sequence from source selection and clinician-informed generation prompts through distillation, decontamination, and medical finetuning. Applying the corpus across several base-model families, while separating fully-open bases from an open-weight control, makes the work more useful than a single-checkpoint release. The described pipeline and associated artifacts could support auditing and controlled follow-up research, even without a new optimization algorithm.
S2. Broad evaluation with substantial attention to failure modes.
The study includes within-base comparisons, multiple-choice and open-ended tasks, multiple judges, order/tie sensitivity, length analyses, component ablations, and a second-teacher comparison. It reports general-capability losses and heterogeneous outcomes across bases rather than presenting adaptation as uniformly beneficial. The paired gains for Apertus and OLMo provide evidence that the training pipeline has practical value within the evaluated settings.
Critical weaknesses
W1. Independent validation of the final training corpus remains limited.
Section 2.3 reports a clinician audit of 60 items from an earlier corpus, followed by pipeline revisions; the final corpus is assessed primarily through LLM screening. On the small earlier audit set, both judges detect only 3 of 6 clinician-identified factual errors and 1 of 4 hallucinations (Appendix B, Table 1), so their final-corpus flags should not be treated as a reliable estimate of residual error. Moreover, pass@8 checks generator-provided labels for synthetic multiple-choice items, while open-ended items have no equivalent label check. Openness and traceability are valuable, but they do not by themselves characterize the factual and safety profile of the released clinical training resource. A modest independent audit of the final version would strengthen this central contribution.
W2. AutoMOOVE validates a narrower notion of clinical preference than the model comparison requires.
The 576 validation pairs were selected for unanimous clinician agreement, and the resulting three-judge agreement is kappa = 0.564 with ties (Section 3.3; Table 13). This is useful validation, but does not establish reliability on disagreement-prone cases or on newly generated MeditronFO/baseline answer pairs. The paper includes meaningful multi-judge and length checks, yet some model comparisons remain sensitive: Gemma-MeditronFO's win rate against MedGemma is 52.8% [49.0, 56.6] under Qwen3.8, and its HealthBench advantage changes sign after the reported length adjustment. These results do not invalidate the within-base gains, but limit the strength of the clinician-preference evidence supporting the new evaluation resource.
Questions for authors
Q1 Has the revised final corpus undergone an independent clinician audit distinct from the original 60-item audit? Please clarify the sampling procedure and residual factual, unsafe-recommendation, and label/rationale-consistency findings. If such evidence is not available, a limited stratified audit of the final synthetic components would materially affect the assessment of the resource.
Q2 Are blinded clinician judgments available for a sample of the newly generated model comparisons, especially MeditronFO versus its base and versus MedGemma, including cases with disagreement? Please distinguish evidence from historical unanimous pairs from evidence supporting the current model rankings. A small targeted comparison would be more informative here than adding another unvalidated judge.
4 comments
Q1: Has the revised final corpus undergone an independent clinician audit distinct from the original 60-item audit? Please clarify the sampling procedure and residual factual, unsafe-recommendation, and label/rationale-consistency findings. If such evidence is not available, a limited stratified audit of the final synthetic components would materially affect the assessment of the resource.
A clinician re-audit of the final corpus is under way. The audit is stratified by component. Clinicians are reviewing 20 randomly sampled items from each of the Guidelines QA and Synthetic MCQA datasets of the final corpus. They rate each item as Wrong, Partly Wrong, Partly Correct, or Correct, and label Wrong and Partly Wrong items as Unsafe recommendation, Factual error, Unclear information or other. We agree that the LLM screening flags should not be read as an estimate of residual error. We now state this in Section 2.3.
Q2: Are blinded clinician judgments available for a sample of the newly generated model comparisons, especially MeditronFO versus its base and versus MedGemma, including cases with disagreement? Please distinguish evidence from historical unanimous pairs from evidence supporting the current model rankings.
A blinded clinician evaluation of the new model comparisons is being conducted with our clinical partners and will be published in a separate study. Meanwhile, we have extended the AutoMOOVE validation beyond unanimous pairs in Appendix I.2, Table 17 in the PDF. On all 1,325 historical pairs with 2+ clinician ratings, including disputed ones, our judge agrees with clinician majority at κ=0.361, compared with κ=0.309 between a clinician and the clinician majority. Agreement holds across the eight model families in the pool.
| Cohen's κ (agreement %) | All pairs (1,325) | Unanimous (576) |
|---|---|---|
| Clinician vs. clinician | 0.223 [0.188, 0.259] (51.4%) | 1 (by construction) |
| Gemma-4-31B vs. each clinician | 0.281 [0.252, 0.311] (55.8%) | 0.593 [0.531, 0.653] (77.0%) |
| Clinician vs. majority of the other clinicians | 0.309 [0.267, 0.350] (58.0%) | 1 (by construction) |
| Gemma-4-31B vs. majority of the other clinicians | 0.361 [0.322, 0.400] (61.6%) | 0.595 [0.532, 0.655] (77.1%) |
By model: (Cohen's κ (agreement %))
| Model | Pairs (all / unanimous) | Clinician vs. other clinicians, all pairs | Gemma-4-31B vs. other clinicians, all pairs | Gemma-4-31B, unanimous pairs |
|---|---|---|---|---|
| Meditron3-70B | 785 / 332 | 0.291 (57.5%) | 0.364 (62.5%) | 0.627 (79.8%) |
| Meditron-DPO | 442 / 201 | 0.338 (60.9%) | 0.449 (68.5%) | 0.671 (82.3%) |
| GPT-4o | 258 / 136 | 0.374 (62.6%) | 0.424 (66.5%) | 0.657 (80.8%) |
| Gemini-2.0-Flash | 219 / 101 | 0.327 (58.5%) | 0.385 (62.7%) | 0.596 (78.0%) |
| MedGemma-27B | 171 / 75 | 0.284 (56.1%) | 0.381 (61.6%) | 0.605 (75.6%) |
| DeepSeek-V3 | 160 / 74 | 0.343 (59.7%) | 0.434 (65.6%) | 0.593 (76.4%) |
| Llama-3.3-70B | 144 / 63 | 0.350 (58.5%) | 0.309 (58.2%) | 0.463 (69.9%) |
| Claude-3.7-Sonnet | 133 / 58 | 0.319 (58.2%) | 0.270 (55.8%) | 0.474 (70.5%) |
| All | 1,325 / 576 | 0.309 (58.0%) | 0.361 (61.6%) | 0.595 (77.1%) |
These pairs do not include MeditronFO outputs, so this supports the judge’s reliability but does not replace clinician review of the new comparisons. We now state in Section 7 (Limitations) that AutoMOOVE win rates are judge preferences, not clinician preferences. As complementary evidence, HealthBench grades the new models’ answers against clinician-written rubrics. On OpenAI’s physician-labelled meta-evaluation (added in Appendix H.1 in the PDF), our grader reaches macro F1 0.698 [0.692, 0.705], against 0.672 for physicians compared with each other.
Q1: Has the revised final corpus undergone an independent clinician audit distinct from the original 60-item audit? Please clarify the sampling procedure and residual factual, unsafe-recommendation, and label/rationale-consistency findings. If such evidence is not available, a limited stratified audit of the final synthetic components would materially affect the assessment of the resource. Followup
In our audit before the pipeline revisions, we got one clinician to review 20 random samples per set. In this audit, we asked 2 clinicians to review 20 distinct random samples per set. Items marked “rejected” in the earlier audit correspond to Wrong or Partly wrong in the current scale.
| Set | Earlier corpus: rejected | Final corpus: Wrong or Partly wrong | Final corpus: Wrong |
|---|---|---|---|
| Synthetic MCQA | 6 / 20 (30%) | 5 / 40 (12.5%) | 3 / 40 (7.5%) |
| Guidelines QA | 2 / 20 (10%) | 5 / 40 (12.5%) | 3 / 40 (7.5%) |
| Both | 8 / 40 (20%, 95% CI 10–35) | 10 / 80 (12.5%, 95% CI 7–22) | 6 / 80 (7.5%, 95% CI 3–15) |
The pipeline revisions show a slight but not statistically significant improvement in the quality of the corpus (p = 0.29). Even after pipeline revisions, the corpus is still limited by the competency of the teacher model. Even if gpt-oss is a competent medical LLM, an estimated 12.5% of sampled items were still flagged as Wrong or Partly wrong, and 7.5% as Wrong. While out of the scope of this submission, we are interested in exploring judge agreement with clinicians in audits of medical training data, as we believe this might show whether LLMs can self correct and find errors in their output.
Additionally the clinicians label for unsafe recommendation went down from 7.5% to 3.8%. But the counts are too small to show a trend.
| Reason | Earlier corpus (40 items) | Final corpus (80 items) |
|---|---|---|
| Ambiguous or unclear question | 6 (15%) | 1 (1.3%) |
| Factual error | 3 (7.5%) | 1 (1.3%) |
| Unsafe recommendation | 3 (7.5%) | 3 (3.8%) |
| Hallucination / not supported by the source | 3 (7.5%) | 1 (1.3%) |
| Important point missing | not asked | 1 (1.3%) |
| Other: answer key disputable | not asked | 2 (2.5%) |
| No reason given | – | 1 (1.3%) |
| Items flagged | 8 (20%) | 10 (12.5%) |
Thank you for the careful and constructive review, and for recognising the value of the resource. We answer your questions below.