acceptodds
Under review as a conference paper at ICLR 2027

Fully Open Meditron: Auditable and Reproducible Development of Clinical LLMs

Abstract

Clinical decision-support LLMs are either proprietary or open-weight, without the training data and development pipelines required for end-to-end reproducibility and auditing. We investigate how much of the gap between fully-open and open-weight clinical specialists is closeable under full-openness constraints. We introduce MeditronFO (Meditron Fully Open), to our knowledge, the first fully-open pipeline for clinical LLM finetuning. The pipeline creates three new synthetic datasets (340k QA pairs) from clinician-authored adversarial vignettes, clinical practice guidelines and public medical QA, using open-weight gpt-oss-120b for rejection-sampling distillation with pass@8, with generation prompts co-authored by a panel of four clinicians. We apply the pipeline to five fully-open base models from three families and one open-weight control (Gemma-3-27B). On the average of five medical benchmarks, Apertus-70B-MeditronFO reaches 54.56%, against 48.35% for Apertus-70B-Instruct and 63.75% for MedGemma-27B, closing 40% of the gap; the Gemma-3-27B-MeditronFO control reaches 58.65%. We also introduce AutoMOOVE, an automated open-ended clinician preference benchmark, whose judgments agree with blinded clinicians' preferences (). The Gemma-3-27B-MeditronFO control is level with or ahead of MedGemma-27B on this benchmark (57.1% win rate under a three-judge majority vote, above parity under two of three judges). We release the data, code and model weights as public artifacts.

Primary area:
datasets and benchmarks
Keywords:
Medical LLMs, Open Source datasets, Clinical evaluation, LLM-as-a-judge, Synthetic data generation, Supervised fine-tuning, Clinical decision support

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SyJpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents MeditronFO, a clinical-LLM development pipeline combining public medical QA, synthetic QA, guideline-derived questions, and synthetic clinical vignettes. The final corpus contains 533,888 examples, with answer distillation, label-based rejection sampling where applicable, decontamination, and audit procedures. The pipeline is applied to five fully-open bases and a Gemma-3-27B open-weight control. Apertus-70B improves from 48.35% to 54.56% on the reported five-benchmark average. The paper also introduces AutoMOOVE: judges are validated against 576 historical answer pairs with unanimous clinician preferences, and new models are compared on the corresponding 516 vignettes. Analyses examine corpus components, teacher choice, judge robustness, response length, and general-capability degradation.

Critical strengths

S1. A useful resource-oriented development contribution.

The manuscript specifies a reproducible sequence from source selection and clinician-informed generation prompts through distillation, decontamination, and medical finetuning. Applying the corpus across several base-model families, while separating fully-open bases from an open-weight control, makes the work more useful than a single-checkpoint release. The described pipeline and associated artifacts could support auditing and controlled follow-up research, even without a new optimization algorithm.

S2. Broad evaluation with substantial attention to failure modes.

The study includes within-base comparisons, multiple-choice and open-ended tasks, multiple judges, order/tie sensitivity, length analyses, component ablations, and a second-teacher comparison. It reports general-capability losses and heterogeneous outcomes across bases rather than presenting adaptation as uniformly beneficial. The paired gains for Apertus and OLMo provide evidence that the training pipeline has practical value within the evaluated settings.

Critical weaknesses

W1. Independent validation of the final training corpus remains limited.

Section 2.3 reports a clinician audit of 60 items from an earlier corpus, followed by pipeline revisions; the final corpus is assessed primarily through LLM screening. On the small earlier audit set, both judges detect only 3 of 6 clinician-identified factual errors and 1 of 4 hallucinations (Appendix B, Table 1), so their final-corpus flags should not be treated as a reliable estimate of residual error. Moreover, pass@8 checks generator-provided labels for synthetic multiple-choice items, while open-ended items have no equivalent label check. Openness and traceability are valuable, but they do not by themselves characterize the factual and safety profile of the released clinical training resource. A modest independent audit of the final version would strengthen this central contribution.

W2. AutoMOOVE validates a narrower notion of clinical preference than the model comparison requires.

The 576 validation pairs were selected for unanimous clinician agreement, and the resulting three-judge agreement is kappa = 0.564 with ties (Section 3.3; Table 13). This is useful validation, but does not establish reliability on disagreement-prone cases or on newly generated MeditronFO/baseline answer pairs. The paper includes meaningful multi-judge and length checks, yet some model comparisons remain sensitive: Gemma-MeditronFO's win rate against MedGemma is 52.8% [49.0, 56.6] under Qwen3.8, and its HealthBench advantage changes sign after the reported length adjustment. These results do not invalidate the within-base gains, but limit the strength of the clinician-preference evidence supporting the new evaluation resource.

Questions for authors

Q1 Has the revised final corpus undergone an independent clinician audit distinct from the original 60-item audit? Please clarify the sampling procedure and residual factual, unsafe-recommendation, and label/rationale-consistency findings. If such evidence is not available, a limited stratified audit of the final synthetic components would materially affect the assessment of the resource.

Q2 Are blinded clinician judgments available for a sample of the newly generated model comparisons, especially MeditronFO versus its base and versus MedGemma, including cases with disagreement? Please distinguish evidence from historical unanimous pairs from evidence supporting the current model rankings. A small targeted comparison would be more informative here than adding another unvalidated judge.

4 comments

Thank you for the careful and constructive review, and for recognising the value of the resource. We answer your questions below.

AuthorsOpenReview ↗

Q1: Has the revised final corpus undergone an independent clinician audit distinct from the original 60-item audit? Please clarify the sampling procedure and residual factual, unsafe-recommendation, and label/rationale-consistency findings. If such evidence is not available, a limited stratified audit of the final synthetic components would materially affect the assessment of the resource.

A clinician re-audit of the final corpus is under way. The audit is stratified by component. Clinicians are reviewing 20 randomly sampled items from each of the Guidelines QA and Synthetic MCQA datasets of the final corpus. They rate each item as Wrong, Partly Wrong, Partly Correct, or Correct, and label Wrong and Partly Wrong items as Unsafe recommendation, Factual error, Unclear information or other. We agree that the LLM screening flags should not be read as an estimate of residual error. We now state this in Section 2.3.

AuthorsOpenReview ↗

Q2: Are blinded clinician judgments available for a sample of the newly generated model comparisons, especially MeditronFO versus its base and versus MedGemma, including cases with disagreement? Please distinguish evidence from historical unanimous pairs from evidence supporting the current model rankings.

A blinded clinician evaluation of the new model comparisons is being conducted with our clinical partners and will be published in a separate study. Meanwhile, we have extended the AutoMOOVE validation beyond unanimous pairs in Appendix I.2, Table 17 in the PDF. On all 1,325 historical pairs with 2+ clinician ratings, including disputed ones, our judge agrees with clinician majority at κ=0.361, compared with κ=0.309 between a clinician and the clinician majority. Agreement holds across the eight model families in the pool.

Cohen's κ (agreement %)All pairs (1,325)Unanimous (576)
Clinician vs. clinician0.223 [0.188, 0.259] (51.4%)1 (by construction)
Gemma-4-31B vs. each clinician0.281 [0.252, 0.311] (55.8%)0.593 [0.531, 0.653] (77.0%)
Clinician vs. majority of the other clinicians0.309 [0.267, 0.350] (58.0%)1 (by construction)
Gemma-4-31B vs. majority of the other clinicians0.361 [0.322, 0.400] (61.6%)0.595 [0.532, 0.655] (77.1%)

By model: (Cohen's κ (agreement %))

ModelPairs (all / unanimous)Clinician vs. other clinicians, all pairsGemma-4-31B vs. other clinicians, all pairsGemma-4-31B, unanimous pairs
Meditron3-70B785 / 3320.291 (57.5%)0.364 (62.5%)0.627 (79.8%)
Meditron-DPO442 / 2010.338 (60.9%)0.449 (68.5%)0.671 (82.3%)
GPT-4o258 / 1360.374 (62.6%)0.424 (66.5%)0.657 (80.8%)
Gemini-2.0-Flash219 / 1010.327 (58.5%)0.385 (62.7%)0.596 (78.0%)
MedGemma-27B171 / 750.284 (56.1%)0.381 (61.6%)0.605 (75.6%)
DeepSeek-V3160 / 740.343 (59.7%)0.434 (65.6%)0.593 (76.4%)
Llama-3.3-70B144 / 630.350 (58.5%)0.309 (58.2%)0.463 (69.9%)
Claude-3.7-Sonnet133 / 580.319 (58.2%)0.270 (55.8%)0.474 (70.5%)
All1,325 / 5760.309 (58.0%)0.361 (61.6%)0.595 (77.1%)

These pairs do not include MeditronFO outputs, so this supports the judge’s reliability but does not replace clinician review of the new comparisons. We now state in Section 7 (Limitations) that AutoMOOVE win rates are judge preferences, not clinician preferences. As complementary evidence, HealthBench grades the new models’ answers against clinician-written rubrics. On OpenAI’s physician-labelled meta-evaluation (added in Appendix H.1 in the PDF), our grader reaches macro F1 0.698 [0.692, 0.705], against 0.672 for physicians compared with each other.

AuthorsOpenReview ↗

Q1: Has the revised final corpus undergone an independent clinician audit distinct from the original 60-item audit? Please clarify the sampling procedure and residual factual, unsafe-recommendation, and label/rationale-consistency findings. If such evidence is not available, a limited stratified audit of the final synthetic components would materially affect the assessment of the resource. Followup

In our audit before the pipeline revisions, we got one clinician to review 20 random samples per set. In this audit, we asked 2 clinicians to review 20 distinct random samples per set. Items marked “rejected” in the earlier audit correspond to Wrong or Partly wrong in the current scale.

SetEarlier corpus: rejectedFinal corpus: Wrong or Partly wrongFinal corpus: Wrong
Synthetic MCQA6 / 20 (30%)5 / 40 (12.5%)3 / 40 (7.5%)
Guidelines QA2 / 20 (10%)5 / 40 (12.5%)3 / 40 (7.5%)
Both8 / 40 (20%, 95% CI 10–35)10 / 80 (12.5%, 95% CI 7–22)6 / 80 (7.5%, 95% CI 3–15)

The pipeline revisions show a slight but not statistically significant improvement in the quality of the corpus (p = 0.29). Even after pipeline revisions, the corpus is still limited by the competency of the teacher model. Even if gpt-oss is a competent medical LLM, an estimated 12.5% of sampled items were still flagged as Wrong or Partly wrong, and 7.5% as Wrong. While out of the scope of this submission, we are interested in exploring judge agreement with clinicians in audits of medical training data, as we believe this might show whether LLMs can self correct and find errors in their output.

Additionally the clinicians label for unsafe recommendation went down from 7.5% to 3.8%. But the counts are too small to show a trend.

ReasonEarlier corpus (40 items)Final corpus (80 items)
Ambiguous or unclear question6 (15%)1 (1.3%)
Factual error3 (7.5%)1 (1.3%)
Unsafe recommendation3 (7.5%)3 (3.8%)
Hallucination / not supported by the source3 (7.5%)1 (1.3%)
Important point missingnot asked1 (1.3%)
Other: answer key disputablenot asked2 (2.5%)
No reason given–1 (1.3%)
Items flagged8 (20%)10 (12.5%)
Reviewer_xFAxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MeditronFO, a pipeline designed to support transparent and reproducible development of fully open medical language models. It constructs training data from public medical question-answer datasets and synthetic examples derived from those datasets, clinical guidelines, and clinician-written vignettes. Fine-tuning improves medical benchmark performance to varying degrees across models and tasks, while reducing general-purpose performance in some models. The paper also introduces AutoMOOVE, which uses LLM judges to compare responses to clinical vignettes and assesses their agreement with clinician preferences. Its main contribution is the proposed release of training data, development code, and model weights, together with an evaluation framework for medical responses.

Critical strengths
  • The paper provides generation prompts, summaries of the filtering process, training hyperparameters, and compute details. If the artifacts are released as described, they could provide a useful baseline for fully open medical LLM development.

  • The paper reports mixed results and limitations, including no statistically significant improvement in average MCQA performance for the EuroLLM and Gemma-3 models, IFEval declines for several models, and smaller HealthBench gains after length adjustment. The authors also attribute much of the observed MMLU-Pro improvement to changes in answer format.

  • The paper documents evaluation sensitivities through AutoMOOVE analyses across judges, tie-handling rules, answer orders, and response lengths. For HealthBench, it compares its scores for the teacher model with published scores and regrades the teacher’s answers using an additional grader.

Critical weaknesses
  • The Gemma-based MeditronFO model shows no statistically significant improvement in average MCQA performance, loses ground after HealthBench length adjustment, and exhibits greater general-purpose degradation than MedGemma relative to the same base. These results limit the evidence for the recipe’s effectiveness across model families.

  • Fine-tuned models produce longer answers, and length adjustment substantially reduces several reported improvements, reversing some into losses. This complicates the interpretation of the raw scores, although it does not establish that longer answers alone caused the gains.

  • Some component ablations show benefits, but the effects vary across datasets, model sizes, and evaluation metrics. The component ablations have limited replication, leaving their robustness to training variability uncertain.

  • Judge agreement is assessed on answer pairs with unanimous clinician preferences, leaving performance on disputed cases untested. Results also show sensitivity to the judge, answer order, and response length. Without clinician evaluation of the newly compared model responses, the reported preferences require cautious interpretation.

Questions for authors
  • Which guideline sources were used, and which source materials, including guideline texts, MOOVE seed vignettes, and clinician preference labels, will be released under what terms? Please also clarify the scope of the “fully open” claim given the pipeline’s reliance on an open-weight teacher.

  • What is the composition of the clinician-rated response pairs before filtering for unanimous preferences, including the models represented? How well do the judges agree with clinicians on disputed pairs, and is clinician evaluation available for the model comparisons reported in this paper?

  • During answer distillation for Guidelines QA, does the teacher receive the source guideline? How is correctness checked beyond agreement with the generated answer label, and was the final corpus re-audited by clinicians after the pipeline revisions?

  • What are the answer-parsing failure rates for base and fine-tuned models on the medical MCQA benchmarks, and are the conclusions robust to alternative answer extraction? Were any hyperparameters or checkpoints selected using medical benchmark performance, and if so, which data splits were used?

13 comments

Thank you for the careful and constructive review, and for recognising the value of our open development pipeline and evaluation analyses. Your comments identify several places where we should clarify the scope of our claims and present the supporting evidence more clearly. This is very helpful. We respond point by point below, and we have already updated the PDF.

AuthorsOpenReview ↗

W1: The Gemma-based MeditronFO model shows no statistically significant improvement in average MCQA performance, loses ground after HealthBench length adjustment, and exhibits greater general-purpose degradation than MedGemma relative to the same base. These results limit the evidence for the recipe’s effectiveness across model families.

We agree that our recipe is limited in its effectiveness across model families, we find that it is more effective on a fully-open base. We believe the medical domain gap between fully-open and open weight models can be partly bridged thanks to our recipe. We have clarified this finding in the PDF.

AuthorsOpenReview ↗

W2: Fine-tuned models produce longer answers, and length adjustment substantially reduces several reported improvements, reversing some into losses.

We agree that it is important to control for length bias and we report the results in Section 5.4 and Table 14. After length adjustment from OpenAI provided evaluation code, the HealthBench gain remains significant for Apertus-70B (+10.0 → +4.6), Apertus-8B (+16.1 → +6.1), OLMo-2-32B (+15.0 → +9.1) and EuroLLM-9B (+15.3 → +8.0), but becomes a loss for EuroLLM-22B (−3.3) and Gemma-3-27B (−1.4). We find that on AutoMOOVE, the judge’s preference is not correlated with answer length for Apertus-70B (Table 19). We have added length-controlled teacher distillation as future work in Section 7.

Reviewer_ori5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an open and auditable pipeline for adapting language models to medicine. It combines existing medical question-answering data with synthetic questions and teacher-generated answers, and evaluates the resulting models through knowledge-based and open-ended assessments. Its main contribution is a reusable data and training resource, together with an automated evaluation protocol validated against historical clinician preferences.

Critical strengths
  1. The paper addresses a useful practical problem of making medical LLM development transparent and reproducible. The documented data sources, generation prompts, filtering procedures, and training configurations provide a concrete foundation for others to reproduce and extend this work.
  2. The experimental effort is substantial. The study covers multiple model families, component ablations at two model sizes, a second teacher, repeated training runs, and checks of judge, answer-order, and length sensitivity. This provides a careful assessment of the recipe and its limitations.
Critical weaknesses
  1. The methodological advance is modest and the value added by the synthetic expansion is only partly established. The pipeline combines existing synthetic generation, rejection sampling, and SFT methods. Table 21 shows specific benefits from Synthetic Curated QA and synthetic vignettes, but these vary across model sizes and metrics, while Curated QA provides the most consistent gains. The experiments establish the usefulness of the overall recipe more clearly than the additional contribution of the new synthetic sources.
  2. Direct clinician validation of the final data and model comparisons remains limited. The item-level clinician audit covers 60 examples from an earlier corpus, while the final corpus is screened by LLMs. AutoMOOVE is validated on 576 historical response pairs selected for unanimous clinician agreement. These checks provide useful evidence, but leave uncertainty about the final corpus’s error rate and clinician agreement with the new model comparisons. The authors should provide a targeted clinician check of the final data.
Questions for authors
  1. Did the authors evaluate distillation on Curated QA alone using the same teacher and a comparable training-token budget? This would help quantify the combined contribution of the synthetic expansion beyond the existing leave-one-component-out results.
  2. Several fine-tuned models show substantial instruction-following losses, and the teacher comparison suggests that these losses can be partly mitigated. How do the authors weigh these trade-offs when recommending models and teacher variants for downstream research, particularly when the medical gains are small?
  3. Did the authors exclude held-out benchmark items from the Curated QA exemplar pool before synthetic question generation, in addition to decontaminating the final corpus?
  4. The method overview and some dense tables are difficult to read at normal viewing size. The authors should enlarge the figure labels and the smallest table text, and improve typographic consistency where feasible.
9 comments

Thank you for the careful and constructive review, and for recognising the value of a transparent and reproducible pipeline for medical model development. Below, we respond to your comments and questions point by point.

AuthorsOpenReview ↗

Q1: Did the authors evaluate distillation on Curated QA alone using the same teacher and a comparable training-token budget? This would help quantify the combined contribution of the synthetic expansion beyond the existing leave-one-component-out results.

Yes. The results of Curated QA alone compared to the full corpus are shown below. As the corpus components differ in size, we did not match the training-token budget. Matching would require subsampling or repeating individual samples. For our training mixture we train on 1 epoch of the full corpus, with equal repetition for each individual sample.

Curated QA alone improves the baseline model, but the full corpus mixture is overall stronger on benchmarks, and importantly recovers more of the IFEval loss. The differences are largest on IFEval and AutoMOOVE, and small on MCQA and HealthBench.

modeltraining dataMCQAIFEvalHB fullAutoMOOVE (win rate against base)
8B-46.570.126-
8Bfull corpus49.838.642.175.3
8BCurated QA only49.033.341.565.7
70B-50.663.439.4-
70Bfull corpus55.942.949.389.1
70BCurated QA only54.130.149.184.6

Thank you for this suggestion, we have added the CuratedQA ablation results in Appendix J in the submission.

AuthorsOpenReview ↗

Q2: Several fine-tuned models show substantial instruction-following losses, and the teacher comparison suggests that these losses can be partly mitigated. How do the authors weigh these trade-offs when recommending models and teacher variants for downstream research, particularly when the medical gains are small?

We agree with you that this is a trade-off that should be discussed, and we now do so in Section 5.2. IFeval loose mode shows that part of the performance drop reflects overfitting on a particular answer format. 

We ablated the teacher choice in Section 5.2 “Second teacher ablation” and Appendix K. The main alternative was Qwen3.8-27B, compared with gpt-oss-120b as a teacher, Qwen3.8 gives much smaller instruction-following losses (IFEval 49.5 vs 38.6 for Apertus-8b), and a non statistically significant difference on HealthBench scores. On medical MCQA under strict parsing it scores 2 points lower for Apertus-70b. 

modelteacherMCQAIFEvalHB fullAutoMOOVE
Apertus-8B-46.570.126.0-
Apertus-8Bgpt-oss-120b49.838.642.175.3
Apertus-8BQwen3.8-27B49.849.542.5-
Apertus-70B-50.663.439.4-
Apertus-70Bgpt-oss-120b55.942.949.389.1
Apertus-70BQwen3.8-27B54.049.750.6-

Part of this gap comes from answer formatting, models trained on Qwen3.8 answers leave 5-8% of MCQA answers unparsed compared with 0.1% for gpt-oss-120b. Under lenient answer extraction, the two teachers are level within confidence intervals. 

MCQA mean, Qwen3.8 teacher − gpt-oss teacherUnparsed, Qwen3.8 / gpt-ossStrict (paper)Lenient
Apertus-8B5.2% / 0.1%+0.0 [−1.1, +1.1]+0.7 [−0.4, +1.8]
Apertus-70B8.4% / 0.1%−1.9 [−3.0, −0.7]+0.5 [−0.6, +1.6]

In our submission we report strict parsing because it keeps results reproducible and comparable across papers, and we use lenient parsing as a robustness check. Under this protocol gpt-oss-120b is the strongest teacher for medical MCQA, so our main MeditronFO corpus is built on it. For downstream research, we recommend the gpt-oss-120b variant when standardized medical benchmark performance is the priority, and the Qwen3.8 variant when instruction following matters. We added this trade-off discussion in Section 5.2 and we will release both variants as Apache 2.0 artifacts for the community.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

Thank you for your constructive reviews. We are glad that the reviewers found the paper addresses "a useful practical problem of making medical LLM development transparent and reproducible" (ori5), is "a useful resource-oriented development contribution" (SyJp) that "could provide a useful baseline for fully open medical LLM development" (xFAx), and offers a "broad evaluation with substantial attention to failure modes" (SyJp), with "substantial" experimental effort (ori5).

We summarize how we addressed the reviewers' concerns below. We have already updated the PDF.

  1. AutoMOOVE validation beyond unanimous pairs (ori5, xFAx, SyJp):
    We have extended the AutoMOOVE validation beyond unanimous pairs (Appendix I.2, table 17). On all 1,325 historical pairs with 2+ clinician ratings, including disputed ones, our judge agrees with the clinician majority at κ=0.361, compared with κ=0.309 for a clinician with the clinician majority.
  2. The value added by the synthetic expansion (ori5, xFAx):
    Our contribution is the open, auditable data and training resource and its evaluation. The new Curated QA-only ablation (Appendix J) quantifies the combined contribution of the synthetic sources: the gains are largest on IFeval and AutoMOOVE, and small on MCQA and HealthBench. Replication results show that the curatedQA ablation exceeds the run to run noise.
  3. The recipe's effectiveness across model families and answer length (xFAx, SyJp):
    We agree that our recipe is limited in its effectiveness across model families, we find that it is more effective on a fully-open base. We believe the medical domain gap between fully-open and open weight models can be partly bridged thanks to our recipe. After length adjustment, the HealthBench gain remains significant for Apertus-70B, Apertus-8B, OLMo-2-32B and EuroLLM-9B, but becomes a loss for EuroLLM-22B and Gemma-3-27B.
  4. A new targeted clinician check of the final data (ori5, xFAx, SyJp):
    In this new audit, we asked 2 clinicians to review 20 distinct random samples per set. It covers the Guidelines QA and Synthetic MCQA datasets. The corpus is still limited by the competency of the teacher model. An estimated 12.5% of sampled items were still flagged as Wrong or Partly wrong, and 7.5% as Wrong. Additionally the clinicians label for unsafe recommendation went down from 7.5% to 3.8%. But the counts are too small to show a trend.
  5. Answer-parsing failure rates and teacher variants (ori5, xFAx):
    The benchmark gains from applying our training mixture persist when using lenient parsing to judge the answers. We added Table 11 in Appendix F. Compared with gpt-oss-120b as a teacher, Qwen3.8 gives much smaller instruction-following losses (IFEval 49.5 vs 38.6 for Apertus-8b), and a non statistically significant difference on HealthBench scores. Under lenient answer extraction, the two teachers are level within confidence intervals. We will release both variants as Apache 2.0 artifacts.
  6. The scope of the "fully open" claim (xFAx):
    By “fully open” we mean that others can inspect the complete training corpus, reproduce our training procedure, and reuse our work. We therefore publicly release the Meditron Fully Open Corpus under Apache 2.0, together with the preprocessing and training code. The corpus includes the teacher-generated answers used for training.
  7. In progress:
    A blinded clinician evaluation of the new model comparisons is being conducted and will be published in a separate study. We are in the process of open sourcing our MOOVE seed vignettes and clinician preference labels as Apache 2.0.

To our knowledge, MeditronFO is the first fully open pipeline for medical finetuning, and it achieves the best results among fully open models on the medical benchmarks we evaluate. We release the data, code and model weights as public artifacts, and we hope this recipe will serve as a stepping stone for fully open research on medical expert models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.